Subscription Operation Instructions
1. The region of the DTS subscription task must be consistent with the region to which the cloud database belongs.
2. After the specified database or collection of the selected subscription object is deleted in the source database, the subscription data (Change Stream) for that database or collection will be invalidated. Even if the database or collection is recreated in the source database, the data subscription cannot be resumed. It is necessary to reset the subscription task and reselect the subscription objects.
Consumption Operation Instructions
1. During data consumption, consumption via public network connection to the Kafka for data subscription is not supported; only access through the Tencent Cloud private network is allowed, and the consumption region must be consistent with the region of the DTS subscription task.
2. To ensure data reentrancy, DTS subscription introduces a Checkpoint mechanism. When messages are written to Kafka Topic, a Checkpoint is generally inserted every 10 seconds to mark the data synchronization checkpoint. After a task interruption and restart, the breakpoint position is identified to enable resumable transfer. Additionally, when the consumer encounters a Checkpoint message, it performs a Kafka consumption offset commit to update the consumption offset in a timely manner.
Built-in Kafka Description
1. The subscribed messages are stored in the DTS built-in Kafka (single-topic). Currently, the default retention period is the most recent 1 day, with a maximum storage capacity of 500G per single-topic. When data storage exceeds 1 day or the data volume surpasses 500G, the built-in Kafka will begin purging the earliest written data. Therefore, users should consume the data promptly to prevent it from being purged before consumption is complete.
2. The built-in Kafka in DTS has a certain upper limit for processing a single message. When a single row of data in the source database exceeds 10MB, this data may be discarded at the consumer side.
Supported SQL Operations for Subscription
|
DML | INSERT,UPDATE,DELETE |
DDL | INDEX:createIndexes,createIndex,dropIndex,dropIndexes
COLLECTION:createCollection,drop,collMod,renameCollection
DATABASE:dropDatabase,copyDatabase |