consumerDemo-avro-java\\src\\main\\resources\\avro-tools-1.8.2.jar: A tool for generating code related to the Avro protocol.consumerDemo-avro-java\\src\\main\\java\\com\\tencent\\subscribe\\avro: The directory where the Avro tool generates code.consumerDemo-avro-java\\src\\main\\resources\\Record.avsc: The protocol definition file.{"namespace": "com.tencent.subscribe.avro", //The last schema in Record.avsc, with "name" displayed as "Record""type": "record","name": "Record", //"name" is displayed as "Record", indicating the format of data consumed from kafka"fields": [{"name": "id", //id indicates a globally incremental ID. For more information about record values, see the following table."type": "long","doc": "unique id of this record in the whole stream"},{"name": "version", //version indicates the protocol version"type": "int","doc": "protocol version"},{"name": "messageType", //Message type"aliases": ["operation"],"type": {"namespace": "com.tencent.subscribe.avro","name": "MessageType","type": "enum","symbols": ["INSERT","UPDATE","DELETE","DDL","BEGIN","COMMIT","HEARTBEAT","CHECKPOINT","ROLLBACK"]}},{...},}
Field Name in Record | Description |
id | Globally incrementing ID. |
version | Protocol version. The current version is 1. |
messageType | Message type. Enum values: "INSERT", "UPDATE", "DELETE", "DDL", "BEGIN", "COMMIT", "HEARTBEAT", "CHECKPOINT". |
fileName | Name of the binlog file where the current record resides. |
position | End offset of the current record in the binlog, in the format of End_log_pos@binlog file number. For example, if the current record is in the file mysql-bin.000004 and its end offset is 2196, the value is "2196@4". |
safePosition | Start offset of the current transaction in the binlog, in the same format as above. |
timestamp | Time when the record is written to the binlog, in unix timestamp format with second-level precision.
The timestamp in the event header corresponding to the transaction recorded in the binlog. Long-running transactions on the source may cause a time difference between timestamp and readerTimestamp, which is normal. |
gtid | Current gtid, for example, c7c98333-6006-11ed-bfc9-b8cef6e1a231:9. |
transactionId | Transaction ID. A transaction ID is generated only for commit events. |
serverId | Source database serverId. To view the source database server_id, refer to SHOW VARIABLES LIKE 'server_id'. |
threadId | Session ID that commits the current transaction. For reference, see SHOW processlist;. |
sourceType | Database type of the source database. The current version supports only MySQL. |
sourceVersion | Source database version. To view the source database version, refer to select version();. |
schemaName | Database name. |
tableName | Table name. |
objectName | Format: database_name.table_name. |
columns | Definitions of columns in the table. |
oldColumns | Data of the row before DML execution. For insert messages, this array is null. The array contains 12 types of elements: Integer, Character, Decimal, Float, Timestamp, DateTime, TimestampWithTimeZone, BinaryGeometry, TextGeometry, BinaryObject, TextObject, and EmptyObject. For details, see the definition in the demo. |
newColumns | Data of the row after DML execution. For delete messages, this array is null. The array contains 12 types of elements: Integer, Character, Decimal, Float, Timestamp, DateTime, TimestampWithTimeZone, BinaryGeometry, TextGeometry, BinaryObject, TextObject, and EmptyObject. For details, see the definition in the demo. |
sql | SQL statement of the DDL. |
executionTime | DDL execution duration, in seconds. |
heartbeatTimestamp | Timestamp of the heartbeat message, in seconds. Only heartbeat messages have this field. |
syncedGtid | GTID set parsed by DTS, in the format of c7c98333-6006-11ed-bfc9-b8cef6e1a231:1-13. |
fakeGtid | Whether it is a constructed fake GTID. If gtid_mode is not enabled, DTS constructs a GTID. |
pkNames | If the table in the source database has a primary key, the DML message carries this parameter; otherwise, it does not. |
readerTimestamp | Time when DTS processes this data, in unix timestamp format with millisecond-level precision. |
tags | |
total | If a message is fragmented, records the total number of fragments. This field is meaningless in the current version (version=1) and is reserved for future extension. |
index | If a message is fragmented, records the index of the current fragment. This field is meaningless in the current version (version=1) and is reserved for future extension. |
File/Directory | Description |
resources/Record.avsc | Tencent Cloud avro data structure definition file (to facilitate conversion to the Alibaba Cloud format, basic types defined in Record-ali.avsc, such as Integer, are borrowed here). |
resources/Record-ali.avsc | Alibaba Cloud avro data structure definition file. |
java/com/ClassConvertor | Utility class for converting Tencent Cloud data structures to Alibaba Cloud data structures. |
java/com/UserBusinessProcess | User business logic class. After Kafka messages are deserialized and converted to the Alibaba Cloud data format, they are passed here, and then certain business logic is executed. This Demo only implements the feature of printing or parsing into SQL statements. Users can extend it here based on business needs, or migrate the business processing code from the previous Alibaba Cloud consumer program here, and compile and run it after simple adaptation. |
Field Name | Type | Description |
version | int | Protocol version. The current version is 2. |
id | long | Globally incrementing Id. |
sourceTimestamp | long | Timestamp of the current record in the binlog event, in seconds. |
sourcePosition | String | End offset of the current record in the binlog, in the format of End_log_pos@binlog file number. For example, if the current record is in the file mysql-bin.000004 and its end offset is 2196, the value is "2196@4". |
safeSourcePosition | String | Start offset of the current transaction in the binlog, in the same format as above. |
sourceTxid | String | Transaction Id. Only COMMIT records carry this field. |
source | com.alibaba.dts.formats.avro.Source | Source database type and version information. |
operation | com.alibaba.dts.formats.avro.Operation | Message type. Supported values: INSERT, UPDATE, DELETE, DDL, BEGIN, COMMIT, ROLLBACK, HEARTBEAT, CHECKPOINT. |
objectName | String | Database_name.table_name. |
processTimestamps | List | Timestamp when the DTS subscription service processes data. Not implemented. |
tags | Map<String,String> | Some additional information. Currently supported keys include thread_id, server_id, pk_uk_info, readerThroughoutTime, GTID, and GTID_SET. |
fields | Object | Definitions of columns in the table, that is, the table header. |
beforeImages | Object | Data of the row before DML execution. For insert messages, this array is null. The array contains 12 types of elements: Integer, Character, Decimal, Float, Timestamp, DateTime, TimestampWithTimeZone, BinaryGeometry, TextGeometry, BinaryObject, TextObject, and EmptyObject. |
afterImages | Object | Data of the row after DML execution. For delete messages, this array is null. The array contains 12 types of elements: Integer, Character, Decimal, Float, Timestamp, DateTime, TimestampWithTimeZone, BinaryGeometry, TextGeometry, BinaryObject, TextObject, and EmptyObject. |
bornTimestamp | long | Not implemented. No corresponding field exists in the Tencent Cloud DTS subscription service. |
MySQL Type | Corresponding Avro Type |
MYSQL_TYPE_NULL | EmptyObject |
MYSQL_TYPE_INT8 | Integer |
MYSQL_TYPE_INT16 | Integer |
MYSQL_TYPE_INT24 | Integer |
MYSQL_TYPE_INT32 | Integer |
MYSQL_TYPE_INT64 | Integer |
MYSQL_TYPE_BIT | Integer |
MYSQL_TYPE_YEAR | DateTime |
MYSQL_TYPE_FLOAT | Float |
MYSQL_TYPE_DOUBLE | Float |
MYSQL_TYPE_VARCHAR | Character |
MYSQL_TYPE_STRING | Character. If the original type is binary, this type will correspond to BinaryObject. |
MYSQL_TYPE_VAR_STRING | Character. If the original type is varbinary, this type will correspond to BinaryObject. |
MYSQL_TYPE_TIMESTAMP | Timestamp |
MYSQL_TYPE_DATE | DateTime |
MYSQL_TYPE_TIME | DateTime |
MYSQL_TYPE_DATETIME | DateTime |
MYSQL_TYPE_TIMESTAMP_NEW | Timestamp |
MYSQL_TYPE_DATE_NEW | DateTime |
MYSQL_TYPE_TIME_NEW | DateTime |
MYSQL_TYPE_DATETIME_NEW | DateTime |
MYSQL_TYPE_ENUM | TextObject |
MYSQL_TYPE_SET | TextObject |
MYSQL_TYPE_DECIMAL | Decimal |
MYSQL_TYPE_DECIMAL_NEW | Decimal |
MYSQL_TYPE_JSON | TextObject |
MYSQL_TYPE_BLOB | BinaryObject |
MYSQL_TYPE_TINY_BLOB | BinaryObject |
MYSQL_TYPE_MEDIUM_BLOB | BinaryObject |
MYSQL_TYPE_LONG_BLOB | BinaryObject |
MYSQL_TYPE_GEOMETRY | BinaryObject |
Esta página foi útil?
Você também pode entrar em contato com a Equipe de vendas ou Enviar um tíquete em caso de ajuda.
comentários