Consuming TDSQL for MySQL Data with Flink

Baixar
Modo Foco
Tamanho da Fonte
Última atualização: 2026-11-05 10:31:38
Traduzido por IA
Overview
In Data Subscription Kafka Edition (with the current Kafka Server version being V2.6.0), you can use a Flink client (only the DataStream API client type is supported) to consume subscription data in Protobuf format. This scenario provides a Demo example of using consumer-demo-tdsql-pb-flink for data consumption, allowing you to quickly test the data consumption process and understand how to parse the data format.
Prerequisites
1. A data consumption task has been created.
2. A consumer group has been created.
3. The Flink runtime environment is installed and can run Flink jobs properly.
Note
The Demo does not include a demonstration of how to consume data. It only prints the data, so you need to write your own data processing logic based on it.
Currently, consuming data by connecting to the Kafka of data subscription over the public network is not supported. Access is supported only through the Tencent Cloud private network, and the region of the subscribed database instance must be the same as the region where data is consumed.
When you subscribe to specified database/table objects (not all objects in the source instance) and use a single Kafka partition, DTS parses the incremental data and writes only the data of the subscribed objects to the Kafka Topic. Data of non-subscribed objects is converted into empty transactions and written to the Kafka Topic, so empty transactions appear during data consumption. The Begin/Commit messages of empty transactions retain the GTID information of the transactions, ensuring the continuity and integrity of GTIDs.
To ensure data reentrancy, DTS subscription introduces a Checkpoint mechanism. When messages are written to the Kafka Topic, a Checkpoint is generally inserted every 10 seconds to mark the data synchronization checkpoint. After a task is interrupted and restarted, the breakpoint position is identified to implement resumable transfer. In addition, when the consumer encounters a Checkpoint message, it commits the Kafka consumption offset once to update the consumption offset in a timely manner.
Consumption Demo Download
For the logic explanation and key parameter descriptions in the Demo, see Protobuf Demo Description (Flink).
When a subscription task is configured, the subscription data format supported by TDSQL MySQL is Protobuf. Protobuf uses a binary format, which offers higher consumption efficiency. The following Demo already includes the Protobuf protocol file, so no additional download is required. If you choose to download the Protobuf protocol file yourself, use Protobuf 3.X for code generation to ensure proper compatibility of the data structure.
Demo Language
Protobuf(TDSQL MySQL)
Java
Address
Steps for Java Flink Demo
Compilation environment: Maven package management tool, JDK8. You can choose a packaging tool as needed. The following uses Maven as an example.
Runtime environment: Tencent CVM (must be in the same region as the subscription instance to access the private network address of the Kafka server), with JRE8 installed.
Operation steps:
1. Download consumer-demo-tdsql-pb-flink.zip and then decompress the file.
2. Go to the decompressed directory. For your convenience, a pom.xml file is already placed in the directory. You need to modify the Flink version to ensure that the Flink cluster version is the same as the Flink version that pom.xml depends on.
3. The ${flink.version} in the following code must be consistent with the Flink version of the cluster.   
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-streaming-java_${scala.binary.version}</artifactId>
    <version>${flink.version}</version>
    <scope>provided</scope>
</dependency>
﻿
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-connector-kafka_${scala.binary.version}</artifactId>
    <version>${flink.version}</version>
    <exclusions>
        <exclusion>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-clients</artifactId>
        </exclusion>
    </exclusions>
</dependency>
﻿
<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-clients_${scala.binary.version}</artifactId>
    <version>${flink.version}</version>
</dependency>
4. Go to the directory where the pom file is located and package the project with Maven. You can also package it directly with IDEA.
5. Package the project with Maven: mvn clean package.
6. For scenarios where the Flink client type is DataStream API, use Flink client commands to submit the job to the Flink cluster and start consumption.
./bin/flink run consumer-demo-tdsql-pb-flink.jar --brokers xxx --topic xxx --group xxx --user xxx --password xxx --trans2sql
brokers is the private network access address of the data subscription Kafka, and topic is the subscription topic of the data subscription task. You can view both on the Subscription Details page.
group, user, and password are the name, account, and password of the consumer group, respectively. You can view them on the Consumption Management page.
trans2sql indicates whether to convert the data into SQL statements. In Java code, carrying this parameter means converting the data into SQL statements, while not carrying it means no conversion.
7. Execute DML statements in the source database and observe the consumption status of the job submitted in Flink.
CREATE TABLE `flink_test` (
  `id` varchar(32) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL,
  `parent_id` varchar(32) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL,
  `user_id` bigint NOT NULL,
  `type` int NOT NULL COMMENT '1: Expense  2: Income',
  `create_time` timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP(3),
  PRIMARY KEY (`id`) USING BTREE
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci ROW_FORMAT=DYNAMIC
8. Observe the consumption status of the job submitted in Flink.
﻿
﻿
﻿
﻿
﻿
View the logs of a specific task on Task Managers.
﻿
﻿
﻿
View the specific Stdout information on Task Managers.
﻿
﻿
﻿
﻿