Can not find the shared library: libhdfs3.so

[iniyanp]

Hi,

I am trying to install hdfs3 using pip in centos. But when I try to use HDFileSystem(), it is throwing Can not find the shared library: libhdfs3.so ERROR.

Steps I followed:

curl -L "https://bintray.com/wangzw/rpm/rpm" -o /etc/yum.repos.d/bintray-wangzw-rpm.repo \
    && yum makecache \
    && yum install -y libhdfs3 libhdfs3-devel \
    && pip install hdfs3

I have tried installing via conda. It works. But unfortunately we can't use conda in our environment for some reasons.

Could anyone help me? Thanks

[gravitywp]

check lbhdfs3.so in LD_LIBRARY_PATH.
or libdhfs3.so version doesn't match hdfs3 needed.
or you could just compile from source https://github.com/ContinuumIO/libhdfs3-downstream

[martindurant]

To diagnose further, you could try running the ctypes load command in the lib module yourself, to see what the exact error was.

[martindurant]

Note that lib/hdfs3 has never been tested via yum by me. Since I see "wangzw" in the URL, I suspect it may be an old build. I recommend using conda for the installation of complicated python-related binary libraries.

[chanansh]

I also have this problem in a mac. fresh install using conda did not help

[martindurant]

Same suggestions as above...
Note that lib/hdfs3 has not changed or been released in some time, but other libraries have moved on.