Creating a MySQL or TDSQL-C MySQL Data Subscription

Baixar

Modo Foco

Tamanho da Fonte

Última atualização: 2026-11-05 10:31:38

Traduzido e Verificado por IA

This scenario describes how to use DTS to create a data subscription task for MySQL or TDSQL-C MySQL.
Prerequisites
Prepare the source database for subscription and ensure that its version meets the requirements. For more information, see databases supported by data subscription.
binlog has been enabled in the source database.
A subscription account has been created in the source database. The account requires the following permissions: REPLICATION CLIENT, REPLICATION SLAVE, PROCESS, and SELECT on all objects.
The specific grant statements are as follows:
create user 'migration account' IDENTIFIED BY 'account password';
grant SELECT, REPLICATION CLIENT,REPLICATION SLAVE,PROCESS on *.* to 'migration account'@'%';
flush privileges;
Restrictions
Subscribed messages are stored in the built-in Kafka (single Topic) of DTS. By default, the retention period is 1 day, and the maximum storage per Topic is 500 GB. When the retention period exceeds 1 day or the data volume exceeds 500 GB, the built-in Kafka starts to purge the earliest written data. Therefore, consume the data promptly to avoid data being purged before it is fully consumed.
The region for data consumption must be the same as the region of the subscription task.
Geometry-related data types are currently not supported. 
During a subscription task, operations such as modifying subscription objects cause a task restart, which may result in duplicate data when users consume data from the Kafka client.
DTS transfers data by the smallest data unit. In incremental data transfer, each marked checkpoint is a data unit. If a data unit has just been transferred when a restart occurs, no data duplication will be caused. If a data unit is still being transferred when a restart occurs, the data unit needs to be pulled again after the restart to ensure data integrity, which will cause data duplication.
For newly created MySQL or TDSQL-C MySQL subscription tasks, if you are concerned about duplicate data, configure deduplication logic when consuming data.
Supported SQL Operations for Subscription
Operation Type
Supported SQL Operations
DML
INSERT,UPDATE,DELETE
DDL
CREATE DATABASE,DROP DATABASE,CREATE TABLE,ALTER TABLE,DROP TABLE,RENAME TABLE
Operation Steps
1. Log in to the DTS console, select Data Subscription in the left sidebar, and click Create Subscription.
2. On the Create Subscription page, select the corresponding configuration, and click Buy Now.
Parameter
Description
Billing Mode
The yearly/monthly subscription and pay-as-you-go modes are supported.
Database
Select based on the specific database type.
Region
If the source database is a TencentDB instance, the region here must be consistent with the region of the source database instance. If the source database is not a TencentDB instance, the region here must be consistent with the region where data is consumed. If there is no special requirement for the data consumption region, select the region closest to the source database.
Edition
Select 2.0.
Specifications
Select a specification based on your business requirements.
Tags
Set tag keys and values for subscription tasks to efficiently manage multiple tasks through tags.
Subscribed Instance Name
Select Name After Creation: The default task name will be the same as the task ID. You can rename the task after the subscription task is created.
Select Name Now and enter the task name in the input box below.
Terms of Service
Make sure to select "I have read and agree to the Cloud Database Service Terms".
3. After the purchase is successful, return to the subscription list, click Configure Subscription in the Operation column to configure the newly purchased subscription. You can use it only after the configuration is completed.
4. On the Configure data subscription page, select the data subscription task. After completing the configuration, click Test Connectivity. After the test is successful and the network is configured, click Save and Go Next.
Data Subscription Task Settings
Configuration Item
Description
Instance Type
Database type selected during creation. It cannot be modified.
We recommend selecting the primary database for subscription, as the subscription service imposes minimal load on the source database.
Access Type
Select based on your scenario. For preparation steps for different access types, see Preparation Overview.
Public Network: The source database is accessible through a public IP address.
Self-Built on CVM: The source database is deployed on Tencent CVM.
Direct Connect: The source database can connect to Tencent Cloud Virtual Private Cloud (VPC) through DC.
VPN Access: The source database can connect to Tencent Cloud VPC through VPN Access.
Database: The source database is a TencentDB instance.
CCN: The source database can connect to Tencent Cloud VPC through CCN.
Public Network
Host address: IP address or domain name of the source database.
Port: port used by the source database.
Self-Built on CVM
CVM instance: The instance ID of the CVM.
Port: port used by the source database.
Direct Connect
VPC-based Direct Connect gateway: Only the VPC-based Direct Connect gateway is supported for the Direct Connect access type. Confirm the network type associated with the gateway.
VPC: Select the VPC and subnet associated with the Direct Connect gateway and VPN gateway.
Host address: IP address or domain name of the source database.
Port: port used by the source database.
VPN Access
VPN gateway: Select the VPN gateway instance for access.
VPC: Select the VPC and subnet associated with the Direct Connect gateway and VPN gateway.
Host address: IP address or domain name of the source database.
Port: port used by the source database.
Database
Cloud database instance: instance ID of the source database.
We recommend subscribing to the primary database because the subscription service places minimal load on the source database.
When the source database is TencentDB for MySQL, you can select read-only or disaster recovery instances. However, to use read-only instances, you need to submit a ticket to apply first. If the source is a single-node TencentDB for MySQL instance (formerly Basic Edition, now discontinued), the access method cannot be "Cloud Database". Instead, use the "Public Network" access method. For a single-node TencentDB for MySQL instance (cloud disk edition), there is no such restriction, and you can use the "Cloud Database" access method directly.
When the source database is TDSQL-C for MySQL, you can select read-only instances.
CCN
Host address: host IP address or domain name of the source database.
Port: port used by the source database.
VPC CCN: Specify the CCN instance name.
Access VPC: The access VPC refers to the VPC in CCN that connects to the subscription link. Select a VPC other than the one to which the source database belongs from all VPCs associated with CCN.
For example, if a database in the Guangzhou region serves as the source database, select a VPC in another region, such as a VPC in Chengdu or Shanghai.
Subnet: name of the subnet in the selected VPC.
Access VPC region: The region of the source database you selected when purchasing the task must be consistent with the region of the access VPC. If they are inconsistent, the source database region will be automatically changed to the VPC access region.
Account/Password
Account and password of the source database.
Connection Method
Secure Sockets layer (SSL) connection refers to the use of SSL to establish a secure connection between DTS and the source database, encrypting the transfer link.
Selecting an SSL secure connection may increase the connection response time of the database. Generally, the private network link of Tencent Cloud is relatively secure, so you do not need to enable SSL secure connection. However, for scenarios where transmission methods such as public network/Direct Connect are used and data security requirements are high, you need to enable SSL secure connection. Before selecting SSL secure connection, you need to enable SSL encryption in the source database first.
Network Settings
Configuration Item
Description
Network
Specify in advance the VPC environment for accessing the Kafka of the subscription task (including the VPC and subnet). You can then consume the data subscription service through this VPC.
Note:
Access within the same VPC: All CVMs in the same VPC can access subscription data normally without additional configuration.
Cross-VPC access: If the consumer CVM is in another VPC, you need to configure CCN to enable cross-VPC access.
5. On the Subscription Type and Object Selection page, select the subscription type, and click Save and Go Next.
Configuration Item
Description
Subscription Type
Data update: Subscribes to data updates for all objects in the source database, including INSERT, UPDATE, and DELETE operations.
Structure update: Subscribes to the creation, modification, and deletion of structures for all objects in the source database.
All instances: Subscribes to data updates and structure updates for all objects in the source database.
Subscription Data Format
Supports three formats: ProtoBuf, Avro, and JSON. ProtoBuf and Avro use a binary format for higher consumption efficiency, while JSON uses a lightweight text format for greater simplicity and ease of use.
Support for eXtended Architecture (XA) transactions
After this option is selected, DTS data subscription can identify XA transaction operations performed in the source database, and data obtained at the consumer side is parsed according to XA transaction logic. If an XA transaction rollback occurs in the source database, DTS can identify the rollback, and the SQL statements before and including the rollback in the source database will not be delivered to the consumer side.
Currently, only data subscription for MySQL and TDSQL-C for MySQL supports XA transactions.
Number of Topic Partitions
Set the number of partitions for topics in the built-in Kafka for data delivery. Increasing the number of partitions can improve the speed of data write operations and consumption. A single partition can guarantee message order, while multiple partitions cannot. If you have strict requirements on the order of consumed messages, select 1 as the number of partitions.
Topic Partition Policy
Partition by table name: Write data with the same table name into the same partition.
Partition by table name + primary key: Data with the same primary key is written to the same partition. This method is suitable for tables with hot data, as it distributes hot table data across different partitions to improve concurrent consumption efficiency.
Use Custom Partition Policy
First, match the database names and table names in the subscription data using regular expressions, and deliver the matched data to partitions by table name, table name + primary key, or column. Then, deliver the remaining unmatched data according to the Topic partition policy. For details, see Setting the Partition Policy.
6. On the pre-check page, the pre-check task is expected to run for 2 to 3 minutes. After the pre-check is passed, click Start to complete the data subscription task configuration.
Note:
If the check fails, see Check Failure Handling to fix the issue and perform the check again.
7. After you click Start, the subscription task will be initialized. This process is expected to take 3 - 4 minutes. After successful initialization, the task enters the Running state.
8. Create Consumer Group: Data Subscription for Kafka allows users to create multiple consumer groups (up to 10 consumer groups per subscription task) for multi-point consumption. Data consumption in Data Subscription for Kafka relies on Kafka consumer groups, so you need to create a consumer group before consuming data. 
9. After the subscription instance enters the Running state, you can start consuming data. Kafka consumption requires password authentication. For specific examples, see Data Consumption Demo, where we provide Demo code in multiple languages and explain the main consumption process and key data structures.
﻿

Ajuda e Suporte

Esta página foi útil?

Você também pode entrar em contato com a Equipe de vendas ou Enviar um tíquete em caso de ajuda.

comentários

tencent cloud

Data Transfer Service

Creating a MySQL or TDSQL-C MySQL Data Subscription

Prerequisites

Restrictions

Supported SQL Operations for Subscription

Operation Steps

Ajuda e Suporte

Operation Type	Supported SQL Operations
DML	INSERT,UPDATE,DELETE
DDL	CREATE DATABASE,DROP DATABASE,CREATE TABLE,ALTER TABLE,DROP TABLE,RENAME TABLE

Parameter	Description
Billing Mode	The yearly/monthly subscription and pay-as-you-go modes are supported.
Database	Select based on the specific database type.
Region	If the source database is a TencentDB instance, the region here must be consistent with the region of the source database instance. If the source database is not a TencentDB instance, the region here must be consistent with the region where data is consumed. If there is no special requirement for the data consumption region, select the region closest to the source database.
Edition	Select 2.0.
Specifications	Select a specification based on your business requirements.
Tags	Set tag keys and values for subscription tasks to efficiently manage multiple tasks through tags.
Subscribed Instance Name	Select Name After Creation: The default task name will be the same as the task ID. You can rename the task after the subscription task is created. Select Name Now and enter the task name in the input box below.
Terms of Service	Make sure to select "I have read and agree to the Cloud Database Service Terms".

Configuration Item	Description
Instance Type	Database type selected during creation. It cannot be modified. We recommend selecting the primary database for subscription, as the subscription service imposes minimal load on the source database.
Access Type	Select based on your scenario. For preparation steps for different access types, see Preparation Overview. Public Network: The source database is accessible through a public IP address. Self-Built on CVM: The source database is deployed on Tencent CVM. Direct Connect: The source database can connect to Tencent Cloud Virtual Private Cloud (VPC) through DC. VPN Access: The source database can connect to Tencent Cloud VPC through VPN Access. Database: The source database is a TencentDB instance. CCN: The source database can connect to Tencent Cloud VPC through CCN.
Public Network	Host address: IP address or domain name of the source database. Port: port used by the source database.
Self-Built on CVM	CVM instance: The instance ID of the CVM. Port: port used by the source database.
Direct Connect	VPC-based Direct Connect gateway: Only the VPC-based Direct Connect gateway is supported for the Direct Connect access type. Confirm the network type associated with the gateway. VPC: Select the VPC and subnet associated with the Direct Connect gateway and VPN gateway. Host address: IP address or domain name of the source database. Port: port used by the source database.
VPN Access	VPN gateway: Select the VPN gateway instance for access. VPC: Select the VPC and subnet associated with the Direct Connect gateway and VPN gateway. Host address: IP address or domain name of the source database. Port: port used by the source database.
Database	Cloud database instance: instance ID of the source database. We recommend subscribing to the primary database because the subscription service places minimal load on the source database. When the source database is TencentDB for MySQL, you can select read-only or disaster recovery instances. However, to use read-only instances, you need to submit a ticket to apply first. If the source is a single-node TencentDB for MySQL instance (formerly Basic Edition, now discontinued), the access method cannot be "Cloud Database". Instead, use the "Public Network" access method. For a single-node TencentDB for MySQL instance (cloud disk edition), there is no such restriction, and you can use the "Cloud Database" access method directly. When the source database is TDSQL-C for MySQL, you can select read-only instances.
CCN	Host address: host IP address or domain name of the source database. Port: port used by the source database. VPC CCN: Specify the CCN instance name. Access VPC: The access VPC refers to the VPC in CCN that connects to the subscription link. Select a VPC other than the one to which the source database belongs from all VPCs associated with CCN. For example, if a database in the Guangzhou region serves as the source database, select a VPC in another region, such as a VPC in Chengdu or Shanghai. Subnet: name of the subnet in the selected VPC. Access VPC region: The region of the source database you selected when purchasing the task must be consistent with the region of the access VPC. If they are inconsistent, the source database region will be automatically changed to the VPC access region.
Account/Password	Account and password of the source database.
Connection Method	Secure Sockets layer (SSL) connection refers to the use of SSL to establish a secure connection between DTS and the source database, encrypting the transfer link. Selecting an SSL secure connection may increase the connection response time of the database. Generally, the private network link of Tencent Cloud is relatively secure, so you do not need to enable SSL secure connection. However, for scenarios where transmission methods such as public network/Direct Connect are used and data security requirements are high, you need to enable SSL secure connection. Before selecting SSL secure connection, you need to enable SSL encryption in the source database first.

Configuration Item	Description
Subscription Type	Data update: Subscribes to data updates for all objects in the source database, including INSERT, UPDATE, and DELETE operations. Structure update: Subscribes to the creation, modification, and deletion of structures for all objects in the source database. All instances: Subscribes to data updates and structure updates for all objects in the source database.
Subscription Data Format	Supports three formats: ProtoBuf, Avro, and JSON. ProtoBuf and Avro use a binary format for higher consumption efficiency, while JSON uses a lightweight text format for greater simplicity and ease of use.
Support for eXtended Architecture (XA) transactions	After this option is selected, DTS data subscription can identify XA transaction operations performed in the source database, and data obtained at the consumer side is parsed according to XA transaction logic. If an XA transaction rollback occurs in the source database, DTS can identify the rollback, and the SQL statements before and including the rollback in the source database will not be delivered to the consumer side. Currently, only data subscription for MySQL and TDSQL-C for MySQL supports XA transactions.
Number of Topic Partitions	Set the number of partitions for topics in the built-in Kafka for data delivery. Increasing the number of partitions can improve the speed of data write operations and consumption. A single partition can guarantee message order, while multiple partitions cannot. If you have strict requirements on the order of consumed messages, select 1 as the number of partitions.
Topic Partition Policy	Partition by table name: Write data with the same table name into the same partition. Partition by table name + primary key: Data with the same primary key is written to the same partition. This method is suitable for tables with hot data, as it distributes hot table data across different partitions to improve concurrent consumption efficiency.
Use Custom Partition Policy	First, match the database names and table names in the subscription data using regular expressions, and deliver the matched data to partitions by table name, table name + primary key, or column. Then, deliver the remaining unmatched data according to the Topic partition policy. For details, see Setting the Partition Policy.