Overview
A MySQL data source can read all or new data from the MySQL database and supports exactly-once semantics. It uses Debezium for change data capture (CDC). It works as follows:
1. Acquires a global read lock to prohibit write operations by other database clients.
2. Starts a repeatable read transaction to ensure that data is always read from the same checkpoint.
3. Reads the current binlog location.
4. Reads the schema of the database and table.
5. Releases the global read lock to allow write operations by other database clients.
6. Scans the table and, after all data is read, obtains the changes made after the binlog location in step 3.
Records the binlog location as checkpoints are created regularly during the execution of the job, so that in case of a crash, processing can resume from the last recorded binlog location. This ensures exactly-once semantics.
Type mapping
|
TINYINT | TINYINT |
SMALLINT | SMALLINT |
TINYINT UNSIGNED |
|
INT | INT |
MEDIUMINT |
|
SMALLINT UNSIGNED |
|
BIGINT | BIGINT |
INT UNSIGNED |
|
BIGINT UNSIGNED | DECIMAL(20, 0) |
FLOAT | FLOAT |
DOUBLE | DOUBLE |
DOUBLE PRECISION |
|
NUMERIC(p, s) | DECIMAL(p, s) |
DECIMAL(p, s) |
|
BOOLEAN | BOOLEAN |
TINYINT(1) |
|
DATE | DATE |
TIME [(p)] | TIME [(p)] [WITHOUT TIMEZONE] |
DATETIME [(p)] | TIMESTAMP [(p)] [WITHOUT TIMEZONE] |
TIMESTAMP [(p)] | TIMESTAMP [(p)] |
TIMESTAMP [(p)] WITH LOCAL TIME ZONE |
|
CHAR(n) | STRING |
VARCHAR(n) |
|
TEXT |
|
BINARY | BYTES |
VARBINARY |
|
BLOB |
|
Notes
User permissions
The user of the source database must have the following permissions: SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT, SELECT, and RELOAD.
Database parameters
The running value of binlog_row_image should be FULL.
WITH parameters
MySQL data sources are developed based on MySQL CDC. The two have identical WITH parameters. For details on how to configure them, see MySQL CDC.