tencent cloud

Data Transfer Service

DokumentasiData Transfer ServiceData Subscription 2.0MySQL Series Data SubscriptionConsuming MySQL DataConsuming Data with FlinkUsing Flink to Consume MySQL or TDSQL-C MySQL Subscription Data

Using Flink to Consume MySQL or TDSQL-C MySQL Subscription Data

Unduh
Mode fokus
Ukuran font
Terakhir diperbarui: 2026-11-05 10:31:38
Diterjemahkan & Diperiksa oleh AI

Overview

In Data Subscription Kafka Edition (with the current Kafka Server version being V2.6.0), you can use a Flink client (only the DataStream API client type is supported) to consume subscription data in Avro format. This scenario provides a Demo example of using flink-dts-connector for data consumption.

Prerequisites

1. A data consumption task has been created.
2. A consumer group has been created.
3. Flink is installed and can run Flink jobs properly.

Note

The Demo does not include a demonstration of how to consume data. It only prints the data, so you need to write your own data processing logic based on it. You can also use Kafka clients in other languages to consume and parse data.
Currently, consuming data by connecting to the Kafka of data subscription over the public network is not supported. Access is supported only through the Tencent Cloud private network, and the region of the subscribed database instance must be the same as the region where data is consumed.
The built-in Kafka in DTS subscription has a certain limit on processing a single message. When a single row of data in the source database exceeds 5 MB, the subscription task may report an error.
When you subscribe to specified database/table objects (not all objects in the source instance) and use a single Kafka partition, DTS parses the incremental data and writes only the data of the subscribed objects to the Kafka Topic. Data of non-subscribed objects is converted into empty transactions and written to the Kafka Topic, so empty transactions appear during data consumption. The Begin/Commit messages of empty transactions retain the GTID information of the transactions, ensuring the continuity and integrity of GTIDs. In addition, multiple empty transactions are compressed to reduce the number of messages.
To ensure data reentrancy, DTS subscription introduces a checkpoint mechanism. When messages are written to Kafka Server, a checkpoint is generally inserted every 10 seconds to mark the data synchronization checkpoint. After a task is interrupted and restarted, the breakpoint position is identified to implement resumable transfer. In addition, when the consumer encounters a checkpoint message, it commits the Kafka consumption offset once to achieve data reentrancy on the consumer side.

Consumption Demo Download

For the logic explanation and key parameter descriptions in the Demo, see Avro Demo Description (Flink).
Demo Language
Avro(MySQL/TDSQL-C MySQL)
Java
Compilation environment: Maven or Gradle package management tool, JDK8. You can choose a packaging tool as needed. The following uses Maven as an example. Runtime environment: Tencent CVM (must be in the same region as the subscription instance to access the private network address of the Kafka server), with JRE8 installed. Operation steps:
1. Download the Java Flink Demo and decompress it.
2. Go to the resources directory and run the following command:
java -jar avro-tools-1.8.2.jar compile -string schema Record.avsc [code generation path]
Note:
If code has already been generated in the demo directory, you can skip this step.
3. Modify the Flink version in the pom.xml file. The version in the following code must be consistent with the Flink version used by the customer. 
<dependency>
          <groupId>org.apache.flink</groupId>
          <artifactId>flink-connector-kafka_${scala.binary.version}</artifactId>
          <version>1.13.6</version>
</dependency>
<dependency>
          <groupId>org.apache.flink</groupId>
          <artifactId>flink-streaming-java_${scala.binary.version}</artifactId>
          <version>1.13.6</version>
          <scope>provided</scope>
</dependency>
<dependency>
          <groupId>org.apache.flink</groupId>
          <artifactId>flink-avro</artifactId>
          <version>1.13.6</version>
</dependency>
4. Go to the directory where the pom file is located and package the project with Maven. You can also package it directly with IDEA.
5. Package the project with Maven: mvn clean package.
6. For scenarios where the Flink client type is DataStream API, use Flink client commands to submit the task and start consumption.
./bin/flink run consumerDemo-avro-flink-1.0-SNAPSHOT.jar --brokers xxx --topic xxx --group xxx --user xxx --password xxx -trans2sql
brokers is the private network access address of the data subscription Kafka, and topic is the subscription topic of the data subscription task. You can view both on the Subscription Details page.
group, user, and password are the name, account, and password of the consumer group, respectively. You can view them on the Consumption Management page.
trans2sql indicates whether to convert the data into SQL statements. In Java code, carrying this parameter means converting the data into SQL statements, while not carrying it means no conversion.
7. Observe consumption.
View running tasks.


View task details.



Topik sebelumnya: JSON Demo DescriptionTopik selanjutnya: Consuming TDSQL for MySQL Data with Flink

Bantuan dan Dukungan

Apakah halaman ini membantu?

masukan

Search
    Enter keywords to search documentation