Domain name for API request: cls.intl.tencentcloudapi.com.
This API is used to create a data processing task.
A maximum of 20 requests can be initiated per second for this API.
The following request parameter list only provides API request parameters and some common parameters. For the complete common parameter list, see Common Request Parameters.
| Parameter Name | Required | Type | Description |
|---|---|---|---|
| Action | Yes | String | Common Params. The value used for this API: CreateDataTransform. |
| Version | Yes | String | Common Params. The value used for this API: 2020-10-16. |
| Region | Yes | String | Common Params. For more information, please see the list of regions supported by the product. |
| FuncType | Yes | Integer | Task type. 1: Specify topic; 2: Dynamic creation. For details, please refer to Creating Processing Task Document (https://www.tencentcloud.com/document/product/614/63940?from_cn_redirect=1). |
| SrcTopicId | Yes | String | Log topic ID -Obtain the log topic Id through Get Log Topic List. |
| Name | Yes | String | Processing task name Name limit -Cannot be an empty string -Cannot contain character ' |
| EtlContent | Yes | String | Processing statement. When FuncType is 2, EtlContent must use log_auto_output. Other reference documents -Create a processing task -Function overview |
| TaskType | Yes | Integer | Processing type. 1: Process preview using random data from the source log topic; 2: Process preview using user-defined test data; 3: Create real processing tasks. |
| DstResources.N | No | Array of DataTransformResouceInfo | Target topic_id and alias of the processing task. This parameter is required when FuncType=1, and not required when FuncType=2. Target topic_id. Obtain the log topic Id through Get Log Topic List. Alias limitation: 1. Cannot be an empty string. 2. Cannot contain the character ' |
| EnableFlag | No | Integer | Task status. Valid values: 1 (enabled) and 2 (disabled). |
| PreviewLogStatistics.N | No | Array of PreviewLogStatistic | Test data for preview processing results Obtain the target log topic ID through Get Log Topic List. |
| BackupGiveUpData | No | Boolean | When FuncType is 2, whether to discard data if the count of dynamically created logsets and log topics exceeds product specification limits, default false. false: Create a fallback logset and log topic and write to fallback topic. true: Discard log data. |
| HasServicesLog | No | Integer | Whether to enable service log shipping. Valid values: 1: disable; 2: enable. |
| DataTransformType | No | Integer | Data processing type. Valid values: 0: standard processing task; 1: pre-processing task. A pre-processing task can process the collected logs and then write them into a log topic. |
| KeepFailureLog | No | Integer | Log status of failed jobs, 1: not retain (default), 2: retain. |
| FailureLogKey | No | String | Field name of a failed log. |
| ProcessFromTimestamp | No | Integer | Specify the start time of data processing, a Unix second-level timestamp. -For any time range within the log topic lifecycle, if it exceeds the lifecycle, only process the part with data within the lifecycle. |
| ProcessToTimestamp | No | Integer | Specify the end time of data processing, a Unix second-level timestamp. -Cannot specify a future time -Leave blank to run constantly |
| TaskId | No | String | Preview a task (TaskType is 1 or 2) that is already created and used the capacity to associate with external data. This parameter is required. Data processing task ID - Search the data processing task list basic information (https://www.tencentcloud.com/document/product/614/72182?from_cn_redirect=1) to get the data processing task ID. |
| DataTransformSqlDataSources.N | No | Array of DataTransformSqlDataSource | Associated data source information |
| EnvInfos.N | No | Array of EnvInfo | Set environment variable |
| Parameter Name | Type | Description |
|---|---|---|
| TaskId | String | Task ID |
| RequestId | String | The unique request ID, generated by the server, will be returned for every request (if the request fails to reach the server for other reasons, the request will not obtain a RequestId). RequestId is required for locating a problem. |
Data processing provides the capacity to filter, clean, mask, enrich, and distribute log data, comparable to the open-source component Logstash. Using the data processing DSL (Domain Specified Language) function, with the underlying layer implemented based on Flink, you can easily handle log stream data.
POST / HTTP/1.1
Host: cls.intl.tencentcloudapi.com
Content-Type: application/json
X-TC-Action: CreateDataTransform
<Common request parameters>
{
"EnableFlag": 1,
"Name": "Data processing task"
"FuncType": 0,
"PreviewLogStatistics": [
{
"LineNum": 0,
"FailReason": "errorcode",
"DstTopicId": "81sdqwe5-e39e-4a1e-b2d4-a778df97d825",
"LogContent": "{\"test\":\"demo\"}",
"Time": "2017-08-08 12:12:12"
}
],
"SrcTopicId": "181d630d-3116-40cf-a12b-597be5e83edd",
"TaskType": 0,
"DstResources": [
{
"TopicId": "adfe19a8-6dad-44a6-9e48-24984ea74108",
"Alias": "alias"
}
],
"EtlContent": "fields_set(\"hello\",\"world\")"
}
{
"Response": {
"TaskId": "a3622556-6402-4942-b4ff-5ae32ec29810",
"RequestId": "6ef60bec-0242-43af-bb20-270359fb54a7"
}
}
TencentCloud API 3.0 integrates SDKs that support various programming languages to make it easier for you to call APIs.
The following only lists the error codes related to the API business logic. For other error codes, see Common Error Codes.
| Error Code | Description |
|---|---|
| FailedOperation | Operation failed. |
| FailedOperation.Timeout | Operation timed out |
| InternalError | Internal error. |
| InvalidParameter | Incorrect parameter. |
| InvalidParameter.CrossAccountConflict | Cross-account identifier error |
| InvalidParameter.DataFromTaskConflict | There is a data processing task conflict. |
| InvalidParameter.InvalidEtlContent | The data processing statement is invalid. |
| MissingParameter | Missing parameter. |
| OperationDenied | Operation denied. |
| OperationDenied.ACLFailed | ACL verification failed. |
| OperationDenied.AccountDestroy | The account has been terminated. |
| OperationDenied.AccountIsolate | The account has overdue payments. |
| OperationDenied.AccountNotExists | The account does not exist. |
| ResourceNotFound | The resource does not exist. |
| ResourceNotFound.TopicNotExist | The log topic does not exist. |
Was this page helpful?
You can also Contact sales or Submit a Ticket for help.
Help us improve! Rate your documentation experience in 5 mins.
Feedback