CDH6 install requires additional /user/yarn directories to deploy

[brittup]

The CDH 6 installer will fail on the deployment of yarn as directories are missing under /user/yarn as it cannot deploy the mr-framework.tar.gz. It does not attempt to create them.

It requires the creation of the following directories under /user/yarn:

/user/yarn/mapreduce
/user/yar/mapreduce/mr-framework

Adding manually and setting the permission on them on Isilon directly resolves this issue.

 
19/12/13 12:31:34 INFO uploader.FrameworkUploader: Target hdfs://isilonsczone-cdh6.foo.com:8020/user/yarn/mapreduce/mr-framework/3.0.0-cdh6.3.2-mr-framework.tar.gz
19/12/13 12:31:36 INFO uploader.FrameworkUploader: Set replication to 1 for path: hdfs://isilonsczone-cdh6.foo.com:8020/user/yarn/mapreduce/mr-framework/3.0.0-cdh6.3.2-mr-framework.tar.gz
19/12/13 12:31:36 INFO uploader.FrameworkUploader: Disabling Erasure Coding for path: hdfs://isilonsczone-cdh6.foo.com:8020/user/yarn/mapreduce/mr-framework/3.0.0-cdh6.3.2-mr-framework.tar.gz
19/12/13 12:31:36 ERROR uploader.FrameworkUploader: Error in execution Component of path not found: /user/yarn/mapreduce/mr-framework/3.0.0-cdh6.3.2-mr-framework.tar.gz
java.io.FileNotFoundException: Component of path not found: /user/yarn/mapreduce/mr-framework/3.0.0-cdh6.3.2-mr-framework.tar.gz
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88)
at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:281)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1182)
at org.apache.hadoop.hdfs.DistributedFileSystem$10.doCall(DistributedFileSystem.java:564)
at org.apache.hadoop.hdfs.DistributedFileSystem$10.doCall(DistributedFileSystem.java:561)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.createNonRecursive(DistributedFileSystem.java:582)
at org.apache.hadoop.hdfs.DistributedFileSystem.access$800(DistributedFileSystem.java:122)
at org.apache.hadoop.hdfs.DistributedFileSystem$HdfsDataOutputStreamBuilder.build(DistributedFileSystem.java:3027)
at org.apache.hadoop.mapred.uploader.FrameworkUploader.beginUpload(FrameworkUploader.java:204)
at org.apache.hadoop.mapred.uploader.FrameworkUploader.buildPackage(FrameworkUploader.java:327)
at org.apache.hadoop.mapred.uploader.FrameworkUploader.run(FrameworkUploader.java:126)
at org.apache.hadoop.mapred.uploader.FrameworkUploader.main(FrameworkUploader.java:612)
Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): Component of path not found: /user/yarn/mapreduce/mr-framework/3.0.0-cdh6.3.2-mr-framework.tar.gz
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1499)

[yongxz]

@brittup What steps you have done manually? Could you write them down here?

[brittup]

when fixing manually, I would just create the required missing directories:

cd to the Isilon access zone Hadoop root directory /user/yarn

create the two directories: mapreduce & mapreduce/mr-framework in the /user/yarn path

set the ownership on the new directories to yarn:yarn

eg:
cd /ifs/cdh6/hadoop/hadoop-root/user/yarn

mkdir mapreduce
mkdir mapreduce/mr-framework

chown -R yarn:yarn mapreduce

[tucked]

It does not attempt to create them.

Does this fail on a vanilla (i.e. non-OneFS) deploy too?

• If so, that seems like an upstream bug.
• If not, do we know why this is specific to OneFS?

edit: If the latter, maybe AMBARI-24893 is related?

[lfrancke]

I believe this is fixed now