Bug? The lib skip directory contents

[timoninmaxim]

Hello, I try to find out a reason of the behaviour. It looks like a bug

hdfs3 was installed from the conda-forge.

So with simple hadoop ls commands:

$ hadoop fs -ls / | grep smartdata
drwxr-xr-x - hdfs supergroup 0 2018-02-16 11:39 /smartdata
$ hadoop fs -ls /smartdata
drwxrwxrwx - hdfs supergroup 0 2018-02-02 11:35 /smartdata/hive
...
$ hadoop fs -ls /smartdata/hive
drwxrwxrwx - hdfs supergroup 0 2018-04-26 09:32 /smartdata/hive/external
...

And this is with hdfs3 lib. It does not show listing of /smartdata, but does it for /smartdata/hive.

$ python -c 'from hdfs3 import HDFileSystem; print HDFileSystem().ls("/")'
[u'//smartdata', ...]
$ python -c 'from hdfs3 import HDFileSystem; print HDFileSystem().ls("/smartdata")'
[]
$ python -c 'from hdfs3 import HDFileSystem; print HDFileSystem().ls("/smartdata/hive")'
[u'/smartdata/hive/external', u'/smartdata/hive/hql', ...]

I saw similar issue here. But it seems it is not permission issue:
https://stackoverflow.com/questions/40405527/python-hdfs3-fails-to-list-non-owned-files

[martindurant]

Hm, honestly I have no idea what might be going on here. Does walk fail to descend into the directories?

[timoninmaxim]

Hello, I have found the bug. hdfs3 does not work correctly with ACL.

$ python -c 'from hdfs3 import HDFileSystem; print HDFileSystem().ls("/user")'
[u'/user/cloudera', u'/user/history', u'/user/hive', u'/user/hue', u'/user/jenkins', u'/user/oozie', u'/user/root', u'/user/spark']
$ hadoop fs -setfacl -m user:jenkins:rwx /user/cloudera
$ python -c 'from hdfs3 import HDFileSystem; print HDFileSystem().ls("/user")'
[]
$ python -c 'from hdfs3 import HDFileSystem; print HDFileSystem().ls("/user/cloudera")'
Traceback (most recent call last):
File "", line 1, in
File "/opt/miniconda/lib/python2.7/site-packages/hdfs3/core.py", line 380, in ls
raise FileNotFoundError(path)
IOError: /user/cloudera

[timoninmaxim]

I found similar issue for libhdfs3-downstream
ContinuumIO/libhdfs3-downstream#4

It is fixed for master branch, but conda-forge goes to concat branch for installing libhdfs3
https://github.com/conda-forge/libhdfs3-feedstock/blob/47aec11797cf29907738a02941690ef81de2fcfd/recipe/build.sh#L3

[martindurant]

Yes, I agree that libhdfs3 should be rereleased, I can try to get to that soon.