Release Notes

Product Introduction

Overview

Use Cases

Node Specification

Cluster Architecture

Purchase Guide

Billing Overview

Purchase Methods

Payment Overdue

Getting Started

Operation Guide

Managing Cluster

Accessing Data Warehouse

Monitoring and Alarming

Access Management

Performance Metrics

TPC-B

Tools and Downloads

Data Ingestion

Importing TencentDB Data Offline with DataX

Syncing Incremental Data from MySQL with DataX

Importing and Exporting COS Data at High Speed with External Table

Syncing EMR Data with External Table

Implementing CDWPG UPSERT with Rule

Data Warehouse Development

Creating Airflow in Cloud

API Documentation

History

Introduction

API Category

Making API Requests

Information Query APIs

Instance APIs

Query APIs

Cluster Management APIs

Cluster Operation APIs

Data Types

Error Codes

Practical Tutorial

Data Warehouse Table Development

Table Distribution Key Selection

Table Storage Format Selection

Table Partition Usage

Extension Usage

Cold Data Backup

Statistics and Space Maintenance

FAQs

Tencent Cloud TCHouse-P Policy

Service Level Agreement

Data Processing And Security Agreement

Importing TencentDB Data Offline with DataX

PDF

Modo Foco

Tamanho da Fonte

Última atualização: 2024-11-27 15:36:05

DataX is an open-source CLI that supports importing full or incremental data from TencentDB to Tencent Cloud TCHouse-P. The tool is developed in Java and uses JDBC to connect the source database to the target database. It can run on Windows and Linux. Install the Java environment before use.
DataX installation:
1. Download the source code here and compile it.
2. Directly use datax-v1.0.4-hashdata.tar.gz, an already compiled version.
The following section introduces DataX modified by HashData, which is more efficient to import data to Tencent Cloud TCHouse-P. Tests show that it can import more than 100,000 entries per second. The following is the configuration file to import data from MySQL to Tencent Cloud TCHouse-P:
{
    "job": {
        "setting": {
            "speed": {
                "channel": 3, 
                "byte": 1048576, 
                "record": 1000
            }, 
            "errorLimit": {
                "record": 2, 
                "percentage": 0.02
            }
        }, 
        "content": [
            {
                "reader": {
                    "name": "mysqlreader", 
                    "parameter": {
                        "username": "****", 
                        "password": "****", 
                        "column": [
                            "*"
                        ], 
                        "splitPk": "id", 
                        "connection": [
                            {
                                "table": [
                                    "test1"
                                ], 
                                "jdbcUrl": [
                                    "jdbc:mysql://***:***/db1?serverTimezone=Asia/Shanghai"
                                ]
                            }
                        ]
                    }
                }, 
                "writer": {
                    "name": "gpdbwriter", 
                    "parameter": {
                        "username": "******", 
                        "password": "******", 
                        "column": [
                            "*"
                        ], 
                        "preSql": [
                            "truncate table test1"
                        ], 
                        "postSql": [
                            "select count(*) from test2"
                        ], 
                        "segment_reject_limit": 0, 
                        "copy_queue_size": 2000, 
                        "num_copy_processor": 1, 
                        "num_copy_writer": 1, 
                        "connection": [
                            {
                                "jdbcUrl": "jdbc:postgresql://****:**/db1", 
                                "table": [
                                    "test1"
                                ]
                            }
                        ]
                    }
                }
            }
        ]
    }
}
Parameter description:
1. The writer should be gpdbwriter. postgresqlwriter can also be used to write data to Tencent Cloud TCHouse-P, with a poor insertion efficiency though.
2. For specific meanings and parameter tuning, see DataX.
3. We recommend you add the serverTimezone=Asia/Shanghai parameter to the JDBC URL of mysqlreader to avoid data inconsistency caused by time zone issues.

Ajuda e Suporte

Esta página foi útil?

Você também pode entrar em contato com a Equipe de vendas ou Enviar um tíquete em caso de ajuda.

comentários

tencent cloud

Tencent Cloud TCHouse-P

Importing TencentDB Data Offline with DataX

Ajuda e Suporte