tencent cloud

Data Transfer Service

ProtoBuf Demo Description

Baixar
Modo Foco
Tamanho da Fonte
Última atualização: 2026-11-05 10:31:38
Traduzido e Verificado por IA

Key Demo Logic Description

Message Production Logic

The following section first briefly explains the message production logic, which helps users understand the consumption logic.
We use Protobuf for serialization, and each language Demo includes a Protobuf definition file. The file defines several key structures: Envelope is the final Kafka message structure, Entry is a single subscription event structure, and Entries is a collection of Entry structures. The relationships between the main data structures are shown below:

The production process is as follows:
1. Pull binlog messages, encoding each binlog event into an Entry structure.
message Entry { //Entry is the structure of a single subscription event. One event is equivalent to one binlog event in MySQL.
Header header = 1; //Event header
Event event = 2; //Event body
}


message Header {
int32 version = 1; //Entry protocol version
SourceType sourceType = 2; //The type of the source database, including MySQL, Oracle, and other types
MessageType messageType = 3; //The type of the message, that is, the type of the Event, including BEGIN, COMMIT, DML, etc.
uint32 timestamp = 4; //The timestamp of the Event in the original binlog
int64 serverId = 5; // The server ID of the source
string fileName = 6; // The file name of the source binlog
uint64 position = 7; // The offset of the event in the source binlog file
string gtid = 8; // The GTID of the current transaction
string schemaName = 9; // The schema affected by the change
string tableName = 10; // The table affected by the change
uint64 seqId = 11; // Globally incremental sequence number
uint64 eventIndex = 12; // If a large event is fragmented, each fragment is numbered from 0. This field is meaningless in the current version and reserved for future extension.
bool isLast = 13; // Whether the current event is the last fragment of the event. If so, the value is true. This field is meaningless in the current version and reserved for future extension.
repeated KVPair properties = 15;
}


message Event {
BeginEvent beginEvent = 1; // The begin event in the binlog
DMLEvent dmlEvent = 2; // The dml event in the binlog
CommitEvent commitEvent = 3; // The commit event in the binlog
DDLEvent ddlEvent = 4; // The ddl event in the binlog
RollbackEvent rollbackEvent = 5; // The rollback event. This field is meaningless in the current version.
HeartbeatEvent heartbeatEvent = 6; // The heartbeat event periodically sent by the source database
CheckpointEvent checkpointEvent = 7; // The checkpoint event added by the subscription backend. One is automatically generated every 10 seconds for Kafka production and consumption offset management.
repeated KVPair properties = 15;
}
2. To reduce the amount of messages, multiple Entry structures are merged into a single structure called Entries, where the Entries.items field represents a sequential list of Entry structures. The number of merged Entry structures is governed by the standard that the size of the Entries after the merge does not exceed the single Kafka message size. For a single event that already exceeds the size limit, it will not be merged, and Entries will contain only a unique Entry.
message Entries {
repeated Entry items = 1; //entry list
}
3. Encode Entries using Protobuf to obtain a binary sequence.
4. Place the binary sequence of Entries into the data field of Envelope. When a single Binlog Event is too large, the binary sequence may exceed the Kafka single message size limit. In this case, we split it into multiple segments, each placed into an Envelope. Envelope.total and Envelope.index record the total number of segments and the sequence number of the current Envelope (starting from 0), respectively.
message Envelope {
int32 version = 1; // The protocol version, which determines how the data content is decoded.
uint32 total = 2;
uint32 index = 3;
bytes data = 4; // The current version is 1, indicating that the data in the data field is the result of PB serialization of Entries.
repeated KVPair properties = 15;
}
5. Encode one or multiple Envelope structures generated in the previous step using Protobuf in sequence, and deliver the Envelope structures to Kafka partitions. Multiple Envelope structures in the same Entries are delivered sequentially to the same partition.

Message Consumption Logic

The following section briefly explains the consumption logic. The demos provided in three languages all follow the same process.
1. Create a Kafka consumer.
2. Start consumption.
3. Consume original messages in sequence, and according to the partition in the message, find the corresponding partitionMsgConsumer object to process the message.
4. The partitionMsgConsumer deserializes the original messages into an Envelope structure.
// Convert the Kafka message Value into an Envelope.
envelope := subscribe.Envelope{}
err := proto.Unmarshal(msg.Value, &envelope)
5. The partitionMsgConsumer consumes one or multiple messages consecutively based on the index and total recorded in the Envelope until Envelope.index equals Envelope.total-1 (see the consumption production logic above, which indicates that a complete Entries has been received).
6. Combine the data fields of multiple consecutive Envelope structures received together in sequence. Decode the combined binary sequence into Entries using Protobuf.
if envelope.Index == 0 {
pmc.completeMsg = envelope
} else {
// Concatenate the binary sequences of the split Entries.
pmc.completeMsg.Data = append(pmc.completeMsg.Data, envelope.Data...)
}
if envelope.Index < envelope.Total-1 {
return nil
}
// Deserialize Envelope.Data into Entries.
entries := subscribe.Entries{}
err = proto.Unmarshal(pmc.completeMsg.Data, &entries)

7. Process each item in Entries.items, printing the original Entry structure or converting it into an SQL statement.
8. When a Checkpoint message is consumed, commit the Kafka offset once. A Checkpoint message is a special message that the subscription backend periodically writes to Kafka, one every 10 seconds.

Database Field Mapping and Storage

This section describes the mapping relationship between database field types and data types defined in the serialization protocol. Field values from the source database (such as MySQL) are stored in the Protobuf protocol using the Data structure shown below.
message Data {
DataType dataType = 1;
string charset = 2; // The encoding type of DataType_STRING, value stored in bv
string sv = 3; // The string value of DataType_INT8/16/32/64/UINT8/16/32/64/Float32/64/DataType_DECIMAL
bytes bv = 4; // The value of DataType_STRING/DataType_BYTES
}
The DataType field represents the type of stored field, with possible enumeration values as shown below.
enum DataType {
NIL = 0; // The value is NULL.
INT8 = 1;
INT16 = 2;
INT32 = 3;
INT64 = 4;
UINT8 = 5;
UINT16 = 6;
UINT32 = 7;
UINT64 = 8;
FLOAT32 = 9;
FLOAT64 = 10;
BYTES = 11;
DECIMAL = 12;
STRING = 13;
NA = 14; // The value does not exist (N/A)
}
The bv field stores the binary representation of STRING and BYTES types, the sv field stores the string representation of INT8/16/32/64/UINT8/16/32/64/DECIMAL types, and the charset field stores the encoding type of STRING.
The mapping relationship between MySQL/TDSQL original type and DataType is as follows (the MYSQL_TYPE_INT8/16/24/32/64 modified by UNSIGNED is mapped to UINT8/16/32/32/64 respectively):
Note:
1. Notes on time zone.
DATE, TIME, and DATETIME types do not support time zone.
The TIMESTAMP type supports time zone. For fields of this type: the system will convert the current time zone to Universal Time Coordinated (UTC) when storing them and convert UTC back to the current time zone when querying them.
Therefore, as shown in the table below, the MYSQL_TYPE_TIMESTAMP and MYSQL_TYPE_TIMESTAMP_NEW fields carry the time zone information, which users can convert as needed when consuming data. (For example, the time format output by DTS is a string with time zone "2021-05-17 07:22:42 +00:00", where "+00:00" indicates the UTC time, and users need to consider time zone information when parsing and converting.)
2. For numeric types from the source MySQL, DTS converts them to string types when writing to Kafka. For FLOAT and DOUBLE types, if the absolute value is less than 1e-4 or greater than or equal to 1e+6 when they are converted to a string, the result is represented in scientific notation. Scientific notation represents floating-point numbers as a mantissa multiplied by a power of 10, which more clearly indicates the magnitude and precise value while saving storage space. If you need to convert the data back to numeric types on the consumer side, pay attention to special handling of floating-point strings represented in scientific notation.
MySQL Field Types (TDSQL supports the same types as MySQL)
Corresponding Protobuf DataType Enumeration Values
MYSQL_TYPE_NULL
NIL
MYSQL_TYPE_INT8
INT8
MYSQL_TYPE_INT16
INT16
MYSQL_TYPE_INT24
INT32
MYSQL_TYPE_INT32
INT32
MYSQL_TYPE_INT64
INT64
MYSQL_TYPE_BIT
INT64
MYSQL_TYPE_YEAR
INT64
MYSQL_TYPE_FLOAT
FLOAT32
MYSQL_TYPE_DOUBLE
FLOAT64
MYSQL_TYPE_VARCHAR
STRING
MYSQL_TYPE_STRING
STRING
MYSQL_TYPE_VAR_STRING
STRING
MYSQL_TYPE_TIMESTAMP
STRING
MYSQL_TYPE_DATE
STRING
MYSQL_TYPE_TIME
STRING
MYSQL_TYPE_DATETIME
STRING
MYSQL_TYPE_TIMESTAMP_NEW
STRING
MYSQL_TYPE_DATE_NEW
STRING
MYSQL_TYPE_TIME_NEW
STRING
MYSQL_TYPE_DATETIME_NEW
STRING
MYSQL_TYPE_ENUM
STRING
MYSQL_TYPE_SET
STRING
MYSQL_TYPE_DECIMAL
DECIMAL
MYSQL_TYPE_DECIMAL_NEW
DECIMAL
MYSQL_TYPE_JSON
BYTES
MYSQL_TYPE_BLOB
BYTES
MYSQL_TYPE_TINY_BLOB
BYTES
MYSQL_TYPE_MEDIUM_BLOB
BYTES
MYSQL_TYPE_LONG_BLOB
BYTES
MYSQL_TYPE_GEOMETRY
BYTES

Ajuda e Suporte

Esta página foi útil?

comentários