tencent cloud

Data Transfer Service

DocumentaçãoData Transfer ServiceData Subscription (Kafka Edition)MySQL Series Data SubscriptionCreating MySQL or TDSQL for MySQL Data Subscription

Creating MySQL or TDSQL for MySQL Data Subscription

Download
Modo Foco
Tamanho da Fonte
Última atualização: 2026-07-29 15:22:42
Traduzido por IA
This document provides instructions for creating a data subscription task for TencentDB for MySQL or TDSQL-C for MySQL using DTS.

Prerequisites

Ensure that the TencentDB instances to be subscribed to are ready and their editions meet the requirements. For details, see Databases Supported by Data Subscription.
Enable binlog on the source instance.
Create a subscription account on the source instance. The account requires the following permissions: REPLICATION CLIENT, REPLICATION SLAVE, PROCESS, and SELECT on all objects. The specific grant statement is as follows:
create user 'migration_account' IDENTIFIED BY 'account_password';
grant SELECT, REPLICATION CLIENT, REPLICATION SLAVE, PROCESS on *.* to 'migration_account'@'%';
flush privileges;

Restrictions

Subscribed messages are stored in the built-in Kafka (single Topic) of DTS. Currently, the default retention period is 1 day, and the maximum storage per Topic is 500 GB. When the data retention period exceeds 1 day or the data volume exceeds 500 GB, the built-in Kafka starts to purge the earliest written data. Therefore, consume the data promptly to avoid data being purged before it is fully consumed.
The region for data consumption must be the same as the region of the subscription instance.
Geometry-related data types are currently not supported.
The message delivery semantics for DTS subscription to Kafka is at least once. Therefore, in special circumstances, the consumed data may contain duplicates. For example, if a subscription task restarts, it pulls Binlog from the source starting from the interrupted point and may pull some extra data, resulting in duplicate message delivery. Operations such as modifying subscription objects or recovering abnormal tasks in the console may also cause message duplication. If your business is sensitive to duplicate data, you need to add deduplication logic in the consumption demo based on your business data.

Supported SQL Operations for Subscription

Operation Type
Supported SQL Operations
DML
INSERT,UPDATE,DELETE
DDL
CREATE DATABASE,DROP DATABASE,CREATE TABLE,ALTER TABLE,DROP TABLE,RENAME TABLE

Operation Steps

1. Log in to the DTS console, select Data Subscription in the left sidebar, and click Create Subscription.
2. On the Create Subscription page, select the corresponding configuration, and click Buy Now.
Billing modes: Yearly/monthly subscription and pay-as-you-go are supported.
Region: If the source database is TencentDB, the region must be the same as that of the source database instance. If the source database is not TencentDB, select the same region as the data consumption region. If there is no specific requirement for the data consumption region, select the region closest to the source database.
Database: Select based on the specific database type.
Version: Select the Kafka edition, which supports direct consumption through Kafka clients.
Subscription Instance Name: Edit the name of the current data subscription instance.
3. After a successful purchase, you will return to the data subscription list. Click Configure Subscription in the Operation column to configure the newly purchased subscription. The subscription can be used only after the configuration is complete.
4. On the Select Instance page, select the corresponding configuration for the source database. After completing this, click Connectivity Test. After the test passes, click Save and Next.
Configuration Item
Description
Instance Type
The database type selected at the time of purchase, which cannot be modified.
It is recommended to select the primary database for subscription. The subscription service exerts minimal pressure on the source database.
Access Type
Select based on your scenario. For preparation steps for different access types, see Preparation Overview.
Public Network: The source database is accessible through a public IP address.
Self-built on CVM: The source database is deployed on Tencent CVM.
Direct Connect: The source database can connect to Tencent Cloud Virtual Private Cloud (VPC) through DC.
VPN access: The source database can connect to Tencent Cloud VPC through VPN Access.
Database: The source database is a TencentDB instance.
CCN: The source database can connect to Tencent Cloud VPC through CCN.
Public Network
Host address: IP address or domain name of the source database.
Port: port used by the source database.
Self-Built on CVM
CVM instance: instance ID of the CVM.
Port: port used by the source database.
Direct Connect
VPC Direct Connect Gateway: Only VPC Direct Connect Gateway is supported for Direct Connect. Confirm the network type associated with the gateway.
VPC: Select the VPC and subnet associated with the VPC Direct Connect Gateway and VPN Gateway.
Host address: IP address or domain name of the source database.
Port: port used by the source database.
VPN Access
VPN gateway: Select the VPN gateway instance for access.
VPC: Select the VPC and subnet associated with the VPC Direct Connect Gateway and VPN Gateway.
Host address: IP address or domain name of the source database.
Port: port used by the source database.
Database
Cloud database instance: instance ID of the source database.
It is recommended to select the primary database for subscription. The subscription service exerts minimal pressure on the source database.
When the source database is TencentDB for MySQL, you can select read-only instances or disaster recovery instances. However, to use a read-only instance, you must first submit a ticket for application. If the source is a TencentDB for MySQL single node (formerly Basic Edition - discontinued), the "cloud database" access method cannot be used. Instead, you can use the "public network" access method. If the source is a TencentDB for MySQL single node (Cloud Disk Edition), there is no restriction, and you can normally use the "cloud database" access method.
When the source database is TDSQL-C for MySQL, you can select read-only instances.
CCN
Host address: host IP address or domain name of the source database.
Port: port used by the source database.
VPC CCN: Specify the CCN instance name.
Access VPC: An access VPC refers to a VPC in a CCN that connects to the subscription link. From all VPCs associated with the CCN, select a VPC other than the one to which the source database belongs. For example, if the source database is located in the Guangzhou region, select a VPC in a different region, such as a Chengdu VPC or a Shanghai VPC.
Subnet: name of the selected VPC subnet.
Access VPC region: The region of the source database you selected when purchasing the task must be consistent with the region of the access VPC. If they are inconsistent, the source database region will be automatically changed to the VPC access region.
Account/Password
Account and password of the source database.
Connection Method
Secure Sockets layer (SSL) connection refers to the use of SSL to establish a secure connection between DTS and the source database, encrypting the transfer link.
Enabling the SSL connection may increase the database connection response time. Generally, the private network link of Tencent Cloud is relatively secure, so you do not need to enable the SSL connection. However, for scenarios that use transmission methods such as public networks or dedicated lines and have high data security requirements, you must enable the SSL connection. Before you select the SSL connection, you must first enable SSL encryption for the source database.
Support for eXtended Architecture (XA) transactions
After this option is selected, DTS data subscription can identify XA transaction operations performed on the source database, and the data obtained at the consumption end is parsed according to the logic of XA transactions. If an XA transaction rollback occurs on the source database, DTS can identify the rollback. SQL statements from the rollback and before the rollback on the source database will not be transmitted to the consumption end.
Currently, only data subscription for MySQL and TDSQL-C for MySQL supports XA transactions.
5. On the Subscription Type and Object Selection page, select a subscription type and click Save and Next.
Configuration Item
Description
Subscription Type
Data updates: This includes data updates for all objects in the source database, covering INSERT, UPDATE, and DELETE operations.
Structural updates: This includes subscribing to the creation, modification, and deletion of structures for all objects in the source database.
Full instance: This subscribes to data updates and structural updates for all objects in the source database.
Subscription Data Format
Supports three formats: ProtoBuf, Avro, and JSON. ProtoBuf and Avro use a binary format, which provides higher consumption efficiency. JSON uses a lightweight text format, making it simpler and easier to use.
Topic Partition Count
Sets the number of partitions for the Topic in the built-in Kafka to which data is delivered. Increasing the number of partitions can improve the speed of data write operations and consumption. A single partition can guarantee message order, while multiple partitions cannot. If you have strict requirements on the order of consumed messages, set the number of partitions to 1.
Topic Partition Policy
Partition by table name: Data with the same table name is written to the same partition.
Partition by table name + primary key: Data with the same primary key is written to the same partition. This method is suitable for tables with hotspot data, as it can distribute the data of hot tables across different partitions, thereby improving concurrent consumption efficiency.
Use Custom Partition Policy
First, match the database names and table names in the subscription data using regular expressions, and deliver the matched data to partitions based on the table name, table name + primary key, or column. Then, deliver the remaining unmatched data according to the Topic partition policy. For details, see Set Partition Policy.
6. On the Pre-verification page, the pre-verification task is expected to run for 2–3 minutes. After the pre-verification is passed, click Start to complete the configuration of the data subscription task.
Note:
If the check fails, follow the Check Failure Handling Method to make corrections and then perform the check again.
7. After you click Start, the subscription task will be initialized, which will take 3–4 minutes. After successful initialization, the task will enter the Running status.
8. Add Consumer Groups. The Data Subscription Kafka edition allows you to create multiple consumer groups for multi-point consumption. Consumption in Data Subscription Kafka edition relies on Kafka consumer groups. Therefore, you must create a consumer group before consuming data.
9. After the subscription instance enters the Running state, you can start consuming data. Kafka consumption requires password authentication. We provide Demo code in multiple languages and also explain the main consumption process and key data structures.
Anterior: Databases Supported by Data SubscriptionPróximo: Creating TDSQL for MySQL Data Subscription

Ajuda e Suporte

Esta página foi útil?

comentários