Overview
The SET statement can be used to adjust some key job parameters. Currently, you can configure most parameters in advanced job parameters. Only a few need to be set using the SET statement. Note
SET is an advanced statement. Improper configuration may cause your job to fail, so use it only when necessary.
Comments (--) are not supported for the SET statement.
A semicolon is required at the end of a SET clause.
Syntax
In a SET clause, string-type values must be quoted with single quotation marks. This is not necessary for Boolean or numeric values.
Example
Idle State Retention Time
For statements with large state, such as GROUP BY and JOIN, Flink offers the Idle State Retention Time mechanism. You can specify the minimum and maximum state retention time to avoid OOM caused by the continuous increase of states. The clause below sets the minimum retention time to 5 hours and the maximum retention time to 6 hours.
Note
The minimum and maximum retention time must be at least 5 minutes apart; otherwise, an error will occur and Flink will ignore the settings.
The format of time units should follow Flink's requirements. For example, you can set the time to 10min, 3h, 3hour, 7day, or 7d. Spaces between numbers and time units are optional.
SET execution.min-idle-state-retention = '5 h';
SET execution.max-idle-state-retention = '6 h';
Checkpoint timeout for SQL jobs
By default, the minimum timeout period for SQL jobs is 10 minutes and the maximum period is twice the checkpoint interval. That is to say, the timeout period for a job whose checkpoint interval is 60 seconds must be at least 10 minutes, and the timeout period for a job whose checkpoint interval is 10 minutes cannot exceed 20 minutes.
To customize a checkpoint timeout period, use the SET clause below.
SET CHECKPOINT_TIMEOUT = '300 s';
Note
The Flink option (advanced parameter) execution.checkpointing.timeout may not take effect for SQL jobs. Please use the SET clause above to configure the timeout period.
Mini-batch
Flink SQL supports mini-batch aggregation, which can significantly increase throughput. Mini-batch may increase delay and is therefore disabled by default. To enable mini-batch, add the following clauses to the edit page of your SQL job (the batch size and latency can be modified but cannot be omitted):
set table.exec.mini-batch.enabled = true;
set table.exec.mini-batch.size = 5000;
set table.exec.mini-batch.allow-latency = '200 ms';