Note:
COSN refers to the Hadoop-COS file system.
Feature Introduction
Elastic MapReduce (EMR) clusters support Cloud Object Storage (COS) standard storage buckets and metadata acceleration buckets, but it is necessary to verify whether the JAR package version complies with the environmental requirements in the EMR environment. If not, you need to update the Hadoop-COS package and cos_api-bundle version. This document mainly introduces how to update the JAR package of the COSN file in EMR.
Directions
Warning:
The following examples show two formats of * including hadoop-temrfs-2.8.5-8.0.2-1.0.6.jar and hadoop-temrfs-1.0.6.jar.
1. If the cluster uses cosranger and * is in the hadoop-temrfs-1.0.6.jar format, submit a ticket for the update. 2. When the cluster does not use cosranger and * is not in the hadoop-temrfs-1.0.6.jar format, perform the following operations for a normal upgrade and update.
3. When * is in the hadoop-temrfs-2.8.5-8.0.2-1.0.6.jar format, you do not need to consider whether the cluster uses cosranger. Perform the following operations for a normal upgrade and update.
Scenario 1
When fs.cosn.impl is equal to com.qcloud.emr.fs.TemrfsHadoopFileSystemAdapter in core-site.xml:
1. Replace /usr/local/service/hadoop/share/hadoop/common/lib/hadoop-temrfs-*.jar with temrfs_hadoop_plugin_network-1.3.jar.
2. In core-site.xml, add the following configuration items:
emr.temrfs.download.md5=5d018fcf087d945ec62e3cf2eefc162
emr.temrfs.download.version=2.8.5-8.2.7-1.0.7
Note:
In the above example: Replace 2.8.5 with your Hadoop version, and replace 8.2.7 with the version of the Hadoop-COS package you need. Ensure that the version is 8.2.7 or later. The cos_api-bundle version will be automatically adapted.
For reference, see:
|
2.8.5 | 8.2.7 | 2.8.5-8.2.7-1.0.7 | 05d018fcf087d945ec62e3cf2eefc162 |
3.1.0 | 8.2.7 | 3.1.0-8.2.7-1.0.8 | 53b4f95d1c44a268f16b009e43db7130 |
Scenario 2
When fs.cosn.impl is equal to org.apache.hadoop.fs.cosnative.NativeCosFileSystem in core-site.xml:
1. Replace /usr/local/service/hadoop/share/hadoop/common/lib/hadoop-temrfs-*.jar with temrfs_hadoop_plugin_network-1.3.jar.
2. In core-site.xml, add the following configuration items:
emr.temrfs.download.md5=5d018fcf087d945ec62e3cf2eefc162
emr.temrfs.download.version=2.8.5-8.2.7-1.0.7 (See Scenario 1)
emr.temrfs.download.region=sh
emr.temrfs.tmp.cache.dir=/data/emr/hdfs/tmp/temrfs