产品动态
产品公告
hadoop-cos-2.x.x-${version}.jar 和 cos_api-bundle-${version}.jar 拷贝到 Datax 解压路径plugin/reader/hdfsreader/libs/以及plugin/writer/hdfswriter/libs/下。CLASS_PATH = ("%s/lib/*:%s/plugin/reader/hdfsreader/libs/*:%s/plugin/writer/hdfswriter/libs/*:.") % (DATAX_HOME, DATAX_HOME, DATAX_HOME)
{"job": {"setting": {"speed": {"byte": 10485760},"errorLimit": {"record": 0,"percentage": 0.02}},"content": [{"reader": {"name": "hdfsreader","parameter": {"path": "testfile","defaultFS": "cosn://examplebucket-1250000000/","column": ["*"],"fileType": "text","encoding": "UTF-8","hadoopConfig": {"fs.cosn.impl": "org.apache.hadoop.fs.CosFileSystem","fs.cosn.userinfo.region": "ap-beijing","fs.cosn.tmp.dir": "/tmp/hadoop_cos","fs.cosn.userinfo.secretId": "COS_SECRETID","fs.cosn.userinfo.secretKey": "COS_SECRETKEY"},"fieldDelimiter": ","}},"writer": {"name": "hdfswriter","parameter": {"path": "/user/hadoop/","fileName": "testfile1","defaultFS": "cosn://examplebucket-1250000000/","column": [{"name": "col","type": "string"},{"name": "col1","type": "string"},{"name": "col2","type": "string"}],"fileType": "text","encoding": "UTF-8","hadoopConfig": {"fs.cosn.impl": "org.apache.hadoop.fs.CosFileSystem","fs.cosn.userinfo.region": "ap-beijing","fs.cosn.tmp.dir": "/tmp/hadoop_cos","fs.cosn.userinfo.secretId": "COS_SECRETID","fs.cosn.userinfo.secretKey": "COS_SECRETKEY"},"fieldDelimiter": ":","writeMode": "append"}}}]}}
cosn://examplebucket-1250000000/。ap-beijing，详情请参见 地域和访问域名。bin/datax.py job/hdfs_job.json
2020-03-09 16:49:59.543 [job-0] INFO JobContainer -[total cpu info] =>averageCpu | maxDeltaCpu | minDeltaCpu-1.00% | -1.00% | -1.00%[total gc info] =>NAME | totalGCCount | maxDeltaGCCount | minDeltaGCCount | totalGCTime | maxDeltaGCTime | minDeltaGCTimePS MarkSweep | 1 | 1 | 1 | 0.024s | 0.024s | 0.024sPS Scavenge | 1 | 1 | 1 | 0.014s | 0.014s | 0.014s2020-03-09 16:49:59.543 [job-0] INFO JobContainer - PerfTrace not enable!2020-03-09 16:49:59.543 [job-0] INFO StandAloneJobContainerCommunicator - Total 2 records, 33 bytes | Speed 3B/s, 0 records/s | Error 0 records, 0 bytes | All Task WaitWriterTime 0.000s | All Task WaitReaderTime 0.033s | Percentage 100.00%2020-03-09 16:49:59.544 [job-0] INFO JobContainer -任务启动时刻 : 2020-03-09 16:49:48任务结束时刻 : 2020-03-09 16:49:59任务总计耗时 : 11s任务平均流量 : 3B/s记录写入速度 : 0rec/s读出记录总数 : 2读写失败总数 : 0
文档反馈