Use Instructions

Download

Mode fokus

Ukuran font

Terakhir diperbarui: 2026-07-29 15:28:56

Diterjemahkan oleh AI

Category
Description
Synchronization Object
1. Only basic tables and views are supported for synchronization; objects such as functions, triggers, and stored procedures are not supported.
2. Associated data objects need to be synchronized at the same time. Otherwise, synchronization will fail.
3. During the incremental synchronization phase, table names in the source database that contain the strings "TDSQLagent" or "tdsql_sub" may be filtered or cause synchronization exceptions. This is because these table names are identical to the temporary table names used by the TDSQL system. Specifically, TDSQLagent is a temporary table used during scaling operations, and the tdsql_sub table is a child table for hash-lish and hash-range sharding. Therefore, it is recommended not to use these types of names for tables intended for synchronization from the source.
Impact on the source database
1. During data synchronization, DTS writes the `__tencentdb__` system database in the source database by using the account that executes the synchronization task to record metadata such as transaction marker IDs. Ensure that the source database has read and write permissions for `__tencentdb__`.
To ensure that subsequent data comparison issues can be located, the `__tencentdb__` database will not be deleted from the source database after the synchronization task is completed.
The `__tencentdb__` system database occupies minimal space, approximately 0.01% to 0.1% of the storage capacity of the source database. For example, for a 50 GB source database, the `__tencentdb__` system database occupies approximately 5 MB–50 MB. It employs a single-threaded, connection-waiting mechanism, thus having a negligible impact on source database performance and not preempting any resources.
2. By default, a lock-free synchronization method is adopted. During the full data export phase, no global lock (FTWRL) is applied to the source database; only table-level locks are applied to tables without primary keys.
Target-side Kafka requirements
1. Currently, only the Professional and Standard editions of Kafka that are purchasable on Tencent Cloud are supported as the target end, and they must be version 1.1.1 or later. The historically discontinued physical machine Standard edition is not supported as the target end.
2. Modify the message retention time and message size limit in the target Kafka.
Set the message retention time to 3 days. Data that exceeds the retention period will be cleared, so users should consume the data within the set time. The message size limit, which is the maximum memory size of a single message that Kafka can receive, must be greater than the maximum value of a single row of data in the source database table. This ensures that all data from the source database can be delivered to Kafka normally.
3. It is recommended that the target Topic be empty. Also, during the sync task, do not perform data write operations in the target Topic selected for synchronization. Otherwise, message disorder and task errors may occur.
Description of data synchronization
1. Only the InnoDB, MyISAM, and TokuDB database engines are supported for synchronization. Tables that use any other database engine are skipped and not synchronized by default.
2. A single sync task can be associated with multiple Topics. However, the same Topic cannot be used by multiple sync tasks simultaneously. Otherwise, data disorder may occur, where the consumed data originates from multiple different tasks. This can also lead to data processing exceptions in scenarios such as task retries, resulting in task errors.
3. During the full synchronization phase, a checkpoint message is inserted into the target Kafka for every 100,000 data records synchronized, marking the current data synchronization position.
4. A single message synchronized by DTS to the target Kafka has a performance limit. It is recommended that a single row of data in the source database not exceed 8 MB. Otherwise, an error may be reported during the synchronization process.
5. If a user is certain that a rename operation will be performed on a table during synchronization and the data is delivered to the target kafka, the topic and partition are matched based on the new table name.
6. During incremental synchronization, if binlog statements in STATEMENT format are generated in the source database, data synchronization will fail.
7. ALTER VIEW statements are not supported. When such a statement is encountered, the task skips it and does not perform synchronization. 
8. Synchronizing the entire source instance is not supported. If a database creation operation (CREATE DATABASE) is performed at the source during the subsequent synchronization task, it is recommended that you manually create the corresponding database object at the target before starting the synchronization task.
9. If the CREATE TABLE or RENAME TABLE operation is performed at the source during a synchronization task, you should select the entire database where the table resides when selecting synchronization objects during the configuration phase. Otherwise, the newly created or renamed tables will not be synchronized to the target.
10. If a sync task is already running and the user modifies the number of partitions in the target Ckafka, the task must be restarted (by first pausing and then resuming it) for data to be delivered to the newly added partitions.
Data consumption description
For the full data that already exists in the source database before the sync task starts, because there is no accurate timestamp for the data write operations, the happenAt field in the consumer Demo displays as 1970-01-01 08:00:00. This timestamp field can be ignored. When incremental data is consumed, the timestamp field displays correctly.
Impact of restart
During a sync task, if a task restart occurs (for example, due to an HA switch in the source database or a restart triggered by a background check for task exceptions), it may cause data synchronized to the target Kafka to become duplicated.
DTS synchronizes data in minimal data units. (In the full migration stage, a chunk of data from a single table object is a minimal unit. In the incremental stage, each marked position is a data unit.) If a restart occurs exactly when a data unit has completed synchronization, no data duplication will occur. However, if a restart occurs while a data unit is still being synchronized, that unit must be resynchronized after the restart, which will cause data duplication.
If you are concerned about duplicate data, configure deduplication logic when consuming data.
Operation Restrictions
Do not perform the following operations during synchronization. Otherwise, they will cause the synchronization task to fail.
1. During the full export phase, do not perform DDL operations that change database or table structures in the source database.
2. Do not modify or delete user information (including usernames, passwords, and permissions) and port numbers in the source database and the target side.
3. Do not perform binlog cleanup operations on the source database.
Data type
1. When TDSQL for MySQL is used as the source database and connected via proxy, using the float data type for floating-point numbers can cause inaccurate data precision during the full migration stage. To preserve data precision, it is recommended to use the double data type. The specific impacts of precision issues during the full migration stage are as follows:
The precision of data synchronization is inconsistent between the full migration stage and the incremental stage.
Using float as a key value results in inconsistent primary key data between the full migration stage and the incremental stage.
2. During the full data synchronization stage, DTS exports data from the source database and imports it to the target Kafka, using the utf8mb4 character set for both operations to avoid garbled characters caused by unrecognized character sets.
3. Geometry-related data types are not supported. Such data will cause task errors.
Transaction
A single transaction cannot contain both DML and DDL statements at the same time. Such transactions will cause task errors.
High availability (HA) switch and scale-out
1. If the source database is a non-global transaction identifier (GTID) database, DTS does not support an HA switch on the source. If a switch is performed on the source TDSQL for MySQL, the DTS incremental synchronization may be interrupted.
2. When DTS connects to TDSQL for MySQL using SET, it does not support TDSQL for MySQL scale-out. When DTS connects to TDSQL for MySQL using Proxy, it supports logical scale-out for TDSQL for MySQL. However, DTS may report errors during horizontal scale-out.
Partitioned table synchronization
1. The syntax for primary/secondary partition tables must comply with specifications. A primary Hash partition table can only be created using the shardkey method.
The key syntax for creating partitioned tables in TDSQL for MySQL is as follows. For detailed syntax, see TDSQL for MySQL Official Documentation.
1.1 Primary partition
Primary Hash partition: shardkey
Primary Range partition: TDSQL_DISTRIBUTED BY RANGE
Primary List partition: TDSQL_DISTRIBUTED BY LIST
1.2 Secondary partition
Primary Hash partition + secondary Range/List partition: shardkey + PARTITION BY RANGE/LIST
Primary Range partition + secondary Range/List partition: TDSQL_DISTRIBUTED BY RANGE + PARTITION BY RANGE/LIST
Primary List partition + secondary Range/List partition: TDSQL_DISTRIBUTED BY LIST + PARTITION BY RANGE/LIST
1.3 New secondary partition
Primary Hash partition + secondary Range/List partition: TDSQL_DISTRIBUTED BY HASH + TDSQL_PARTITION BY RANGE/LIST
2. During the incremental synchronization phase, concurrent DDL operations are not supported. You must wait for the previous DDL operation to take effect before executing the next one. Otherwise, DDL operations executed out of order may cause errors. Additionally, rapidly performing create, drop, and create operations on tables with the same name is not supported, as this may lead to table type errors.
Specify Start Checkpoint
If Initialization Type is not specified, it indicates that only incremental synchronization is performed, and you can set the start checkpoint for incremental synchronization. Currently, you need to submit a ticket to apply for using this feature.
1. The incremental synchronization task starts synchronizing data from the set time point. The time point can be set within the range from the previous 14 days to the current time.
2. If full and incremental migrations are performed as separate tasks, ensure that no DDL operations (such as CREATE TABLE, ALTER TABLE, DROP TABLE, RENAME TABLE, CREATE INDEX, and DROP INDEX) occur between time point 1, when the full migration completes, and the set time point 2 for incremental migration. Otherwise, the task will report an error.
3. From the set time point 1 to the start point 2 of the incremental task (the task is considered started only after the task step changes from searching for the specified checkpoint to synchronizing increments). The must-knows for operations on the source database are as follows.
It is recommended that you do not perform a primary-secondary switch, add shards, or recreate secondary servers on the source database; otherwise, it may affect the ability of DTS to obtain the GTID checkpoint of the source database.
You can perform DDL operations that are not related to the synchronization objects on the source. However, ensure that the previous DDL operation has been synchronized to the target before you execute a new one. Executing DDL operations too frequently may cause task errors.
You cannot perform DDL operations related to the synchronization objects (such as CREATE TABLE, ALTER TABLE, DROP TABLE, RENAME TABLE, CREATE INDEX, and DROP INDEX) on the source; otherwise, the synchronization task will report an error.
4. Point-in-time synchronization determines the GTID based on the timestamp specified in the binlog context (SET TIMESTAMP=XXXX). To ensure data synchronization accuracy, it is recommended that users not modify this context.
5. Ensure that the time zone set in the database is consistent with the current console time zone (that is, the browser time zone), or convert the time to the database time zone. Otherwise, the synchronization results for the specified checkpoint may differ from expectations.
6. Ensure that the time is consistent across all cluster nodes in TDSQL for MySQL; otherwise, the synchronization results for the specified checkpoint may differ from expectations.
7. If the set time point falls within the duration of an XA transaction, that XA transaction will not be synchronized. In this case, it is recommended to set the time point earlier (before the XA START time).
8. When the point-in-time feature is used, gtid_mode must be set to ON on the source.
Column filtering
1. Primary key and unique key columns do not support filtering. The system will synchronize primary key and unique key columns to the target database by default. In special cases, if primary keys or unique keys are filtered, a precheck error will be reported.
2. If the filtered columns are composite primary keys, foreign keys, or partition columns, or the filtered columns contain indexes, DML operations on these columns are not supported in the source database. Such operations may cause task errors.
3. The interaction between column filtering and views is described as follows:
3.1 If views are selected as the synchronization objects, the data related to view columns should be synchronized together and cannot be filtered. Otherwise, a task error will be reported.
3.2 If views are selected as the synchronization objects and the views created in the source database are subsequently associated with system database tables, column filtering cannot be used in the synchronization task. Otherwise, a task error will be reported.
4. After column filtering is set, synchronization configuration cannot be modified. When the synchronization configuration is modified, column filtering cannot be set.
5. If column X in table A is filtered and a user executes the CREATE TABLE table_B like table_A statement in the source database, it is recommended to also filter column X in table B. In this scenario, DTS synchronizes table B based on the table structure of table A after column filtering. If column X in table B is not filtered, the sync task will report an error when the source database subsequently updates data in column X of table B.
6. If the unique key column in a table object contains a function and the table is selected as the synchronization object, the table cannot be expanded to the column dimension, and its columns cannot be filtered either.

Bantuan dan Dukungan

Apakah halaman ini membantu?

Anda juga dapat Menghubungi Penjualan atau Mengirimkan Tiket untuk meminta bantuan.

masukan

tencent cloud

Data Transfer Service

Use Instructions

Bantuan dan Dukungan

Category	Description
Synchronization Object	1. Only basic tables and views are supported for synchronization; objects such as functions, triggers, and stored procedures are not supported. 2. Associated data objects need to be synchronized at the same time. Otherwise, synchronization will fail. 3. During the incremental synchronization phase, table names in the source database that contain the strings "TDSQLagent" or "tdsql_sub" may be filtered or cause synchronization exceptions. This is because these table names are identical to the temporary table names used by the TDSQL system. Specifically, TDSQLagent is a temporary table used during scaling operations, and the tdsql_sub table is a child table for hash-lish and hash-range sharding. Therefore, it is recommended not to use these types of names for tables intended for synchronization from the source.
Impact on the source database	1. During data synchronization, DTS writes the `__tencentdb__` system database in the source database by using the account that executes the synchronization task to record metadata such as transaction marker IDs. Ensure that the source database has read and write permissions for `__tencentdb__`. To ensure that subsequent data comparison issues can be located, the `__tencentdb__` database will not be deleted from the source database after the synchronization task is completed. The `__tencentdb__` system database occupies minimal space, approximately 0.01% to 0.1% of the storage capacity of the source database. For example, for a 50 GB source database, the `__tencentdb__` system database occupies approximately 5 MB–50 MB. It employs a single-threaded, connection-waiting mechanism, thus having a negligible impact on source database performance and not preempting any resources. 2. By default, a lock-free synchronization method is adopted. During the full data export phase, no global lock (FTWRL) is applied to the source database; only table-level locks are applied to tables without primary keys.
Target-side Kafka requirements	1. Currently, only the Professional and Standard editions of Kafka that are purchasable on Tencent Cloud are supported as the target end, and they must be version 1.1.1 or later. The historically discontinued physical machine Standard edition is not supported as the target end. 2. Modify the message retention time and message size limit in the target Kafka. Set the message retention time to 3 days. Data that exceeds the retention period will be cleared, so users should consume the data within the set time. The message size limit, which is the maximum memory size of a single message that Kafka can receive, must be greater than the maximum value of a single row of data in the source database table. This ensures that all data from the source database can be delivered to Kafka normally. 3. It is recommended that the target Topic be empty. Also, during the sync task, do not perform data write operations in the target Topic selected for synchronization. Otherwise, message disorder and task errors may occur.
Description of data synchronization	1. Only the InnoDB, MyISAM, and TokuDB database engines are supported for synchronization. Tables that use any other database engine are skipped and not synchronized by default. 2. A single sync task can be associated with multiple Topics. However, the same Topic cannot be used by multiple sync tasks simultaneously. Otherwise, data disorder may occur, where the consumed data originates from multiple different tasks. This can also lead to data processing exceptions in scenarios such as task retries, resulting in task errors. 3. During the full synchronization phase, a checkpoint message is inserted into the target Kafka for every 100,000 data records synchronized, marking the current data synchronization position. 4. A single message synchronized by DTS to the target Kafka has a performance limit. It is recommended that a single row of data in the source database not exceed 8 MB. Otherwise, an error may be reported during the synchronization process. 5. If a user is certain that a rename operation will be performed on a table during synchronization and the data is delivered to the target kafka, the topic and partition are matched based on the new table name. 6. During incremental synchronization, if binlog statements in STATEMENT format are generated in the source database, data synchronization will fail. 7. ALTER VIEW statements are not supported. When such a statement is encountered, the task skips it and does not perform synchronization. 8. Synchronizing the entire source instance is not supported. If a database creation operation (CREATE DATABASE) is performed at the source during the subsequent synchronization task, it is recommended that you manually create the corresponding database object at the target before starting the synchronization task. 9. If the CREATE TABLE or RENAME TABLE operation is performed at the source during a synchronization task, you should select the entire database where the table resides when selecting synchronization objects during the configuration phase. Otherwise, the newly created or renamed tables will not be synchronized to the target. 10. If a sync task is already running and the user modifies the number of partitions in the target Ckafka, the task must be restarted (by first pausing and then resuming it) for data to be delivered to the newly added partitions.
Data consumption description	For the full data that already exists in the source database before the sync task starts, because there is no accurate timestamp for the data write operations, the happenAt field in the consumer Demo displays as 1970-01-01 08:00:00. This timestamp field can be ignored. When incremental data is consumed, the timestamp field displays correctly.
Impact of restart	During a sync task, if a task restart occurs (for example, due to an HA switch in the source database or a restart triggered by a background check for task exceptions), it may cause data synchronized to the target Kafka to become duplicated. DTS synchronizes data in minimal data units. (In the full migration stage, a chunk of data from a single table object is a minimal unit. In the incremental stage, each marked position is a data unit.) If a restart occurs exactly when a data unit has completed synchronization, no data duplication will occur. However, if a restart occurs while a data unit is still being synchronized, that unit must be resynchronized after the restart, which will cause data duplication. If you are concerned about duplicate data, configure deduplication logic when consuming data.
Operation Restrictions	Do not perform the following operations during synchronization. Otherwise, they will cause the synchronization task to fail. 1. During the full export phase, do not perform DDL operations that change database or table structures in the source database. 2. Do not modify or delete user information (including usernames, passwords, and permissions) and port numbers in the source database and the target side. 3. Do not perform binlog cleanup operations on the source database.
Data type	1. When TDSQL for MySQL is used as the source database and connected via proxy, using the float data type for floating-point numbers can cause inaccurate data precision during the full migration stage. To preserve data precision, it is recommended to use the double data type. The specific impacts of precision issues during the full migration stage are as follows: The precision of data synchronization is inconsistent between the full migration stage and the incremental stage. Using float as a key value results in inconsistent primary key data between the full migration stage and the incremental stage. 2. During the full data synchronization stage, DTS exports data from the source database and imports it to the target Kafka, using the utf8mb4 character set for both operations to avoid garbled characters caused by unrecognized character sets. 3. Geometry-related data types are not supported. Such data will cause task errors.
Transaction	A single transaction cannot contain both DML and DDL statements at the same time. Such transactions will cause task errors.
High availability (HA) switch and scale-out	1. If the source database is a non-global transaction identifier (GTID) database, DTS does not support an HA switch on the source. If a switch is performed on the source TDSQL for MySQL, the DTS incremental synchronization may be interrupted. 2. When DTS connects to TDSQL for MySQL using SET, it does not support TDSQL for MySQL scale-out. When DTS connects to TDSQL for MySQL using Proxy, it supports logical scale-out for TDSQL for MySQL. However, DTS may report errors during horizontal scale-out.
Partitioned table synchronization	1. The syntax for primary/secondary partition tables must comply with specifications. A primary Hash partition table can only be created using the shardkey method. The key syntax for creating partitioned tables in TDSQL for MySQL is as follows. For detailed syntax, see TDSQL for MySQL Official Documentation. 1.1 Primary partition Primary Hash partition: shardkey Primary Range partition: TDSQL_DISTRIBUTED BY RANGE Primary List partition: TDSQL_DISTRIBUTED BY LIST 1.2 Secondary partition Primary Hash partition + secondary Range/List partition: shardkey + PARTITION BY RANGE/LIST Primary Range partition + secondary Range/List partition: TDSQL_DISTRIBUTED BY RANGE + PARTITION BY RANGE/LIST Primary List partition + secondary Range/List partition: TDSQL_DISTRIBUTED BY LIST + PARTITION BY RANGE/LIST 1.3 New secondary partition Primary Hash partition + secondary Range/List partition: TDSQL_DISTRIBUTED BY HASH + TDSQL_PARTITION BY RANGE/LIST 2. During the incremental synchronization phase, concurrent DDL operations are not supported. You must wait for the previous DDL operation to take effect before executing the next one. Otherwise, DDL operations executed out of order may cause errors. Additionally, rapidly performing create, drop, and create operations on tables with the same name is not supported, as this may lead to table type errors.
Specify Start Checkpoint	If Initialization Type is not specified, it indicates that only incremental synchronization is performed, and you can set the start checkpoint for incremental synchronization. Currently, you need to submit a ticket to apply for using this feature. 1. The incremental synchronization task starts synchronizing data from the set time point. The time point can be set within the range from the previous 14 days to the current time. 2. If full and incremental migrations are performed as separate tasks, ensure that no DDL operations (such as CREATE TABLE, ALTER TABLE, DROP TABLE, RENAME TABLE, CREATE INDEX, and DROP INDEX) occur between time point 1, when the full migration completes, and the set time point 2 for incremental migration. Otherwise, the task will report an error. 3. From the set time point 1 to the start point 2 of the incremental task (the task is considered started only after the task step changes from searching for the specified checkpoint to synchronizing increments). The must-knows for operations on the source database are as follows. It is recommended that you do not perform a primary-secondary switch, add shards, or recreate secondary servers on the source database; otherwise, it may affect the ability of DTS to obtain the GTID checkpoint of the source database. You can perform DDL operations that are not related to the synchronization objects on the source. However, ensure that the previous DDL operation has been synchronized to the target before you execute a new one. Executing DDL operations too frequently may cause task errors. You cannot perform DDL operations related to the synchronization objects (such as CREATE TABLE, ALTER TABLE, DROP TABLE, RENAME TABLE, CREATE INDEX, and DROP INDEX) on the source; otherwise, the synchronization task will report an error. 4. Point-in-time synchronization determines the GTID based on the timestamp specified in the binlog context (SET TIMESTAMP=XXXX). To ensure data synchronization accuracy, it is recommended that users not modify this context. 5. Ensure that the time zone set in the database is consistent with the current console time zone (that is, the browser time zone), or convert the time to the database time zone. Otherwise, the synchronization results for the specified checkpoint may differ from expectations. 6. Ensure that the time is consistent across all cluster nodes in TDSQL for MySQL; otherwise, the synchronization results for the specified checkpoint may differ from expectations. 7. If the set time point falls within the duration of an XA transaction, that XA transaction will not be synchronized. In this case, it is recommended to set the time point earlier (before the XA START time). 8. When the point-in-time feature is used, gtid_mode must be set to ON on the source.
Column filtering	1. Primary key and unique key columns do not support filtering. The system will synchronize primary key and unique key columns to the target database by default. In special cases, if primary keys or unique keys are filtered, a precheck error will be reported. 2. If the filtered columns are composite primary keys, foreign keys, or partition columns, or the filtered columns contain indexes, DML operations on these columns are not supported in the source database. Such operations may cause task errors. 3. The interaction between column filtering and views is described as follows: 3.1 If views are selected as the synchronization objects, the data related to view columns should be synchronized together and cannot be filtered. Otherwise, a task error will be reported. 3.2 If views are selected as the synchronization objects and the views created in the source database are subsequently associated with system database tables, column filtering cannot be used in the synchronization task. Otherwise, a task error will be reported. 4. After column filtering is set, synchronization configuration cannot be modified. When the synchronization configuration is modified, column filtering cannot be set. 5. If column X in table A is filtered and a user executes the CREATE TABLE table_B like table_A statement in the source database, it is recommended to also filter column X in table B. In this scenario, DTS synchronizes table B based on the table structure of table A after column filtering. If column X in table B is not filtered, the sync task will report an error when the source database subsequently updates data in column X of table B. 6. If the unique key column in a table object contains a function and the table is selected as the synchronization object, the table cannot be expanded to the column dimension, and its columns cannot be filtered either.