Segfault When Initiating HDFS Connection

[lucashu1]

Hi,

I'm getting a segfault when trying to create a connection using the HDFileSystem constructor. The code that I'm running is:

from hdfs3 import HDFileSystem
hdfs = HDFileSystem([HOSTNAME], port=[PORT-NUM])

(with HOSTNAME and PORT-NUM filled in, ofc.)

When I run the script through GDB (for debugging purposes), I get:

Starting program: /home/jovyan/.conda/bin/python hdfs-test.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
strlen () at ../sysdeps/x86_64/strlen.S:106
106	../sysdeps/x86_64/strlen.S: No such file or directory.
(gdb) backtrace
#0  strlen () at ../sysdeps/x86_64/strlen.S:106
#1  0x00007ffff3ff83ee in std::char_traits<char>::length (__s=0x1 <error: Cannot access memory at address 0x1>)
   from /home/jovyan/.conda/lib/python3.6/lib-dynload/../../libhdfs3.so
#2  std::string::assign (__s=0x1 <error: Cannot access memory at address 0x1>, this=0x555555c27fa0)
    at /feedstock_root/build_artefacts/libhdfs3_1526441785297/work/libhdfs3/src/client/FileSystem.cpp:1131
#3  std::string::operator= (__s=0x1 <error: Cannot access memory at address 0x1>, this=0x555555c27fa0)
    at /feedstock_root/build_artefacts/libhdfs3_1526441785297/work/libhdfs3/src/client/FileSystem.cpp:555
#4  Hdfs::FileSystem::FileSystem (this=0x555555c27fa0, conf=..., euser=0x1 <error: Cannot access memory at address 0x1>)
    at /feedstock_root/build_artefacts/libhdfs3_1526441785297/work/libhdfs3/src/client/FileSystem.cpp:148
#5  0x00007ffff400f117 in hdfsBuilderConnect (bld=0x555555bf8c40, 
    effective_user=0x1 <error: Cannot access memory at address 0x1>)
   from /home/jovyan/.conda/lib/python3.6/lib-dynload/../../libhdfs3.so
#6  0x00007ffff6607ec0 in ffi_call_unix64 () from /home/jovyan/.conda/lib/python3.6/lib-dynload/../../libffi.so.6
#7  0x00007ffff660787d in ffi_call () from /home/jovyan/.conda/lib/python3.6/lib-dynload/../../libffi.so.6
#8  0x00007ffff681cdee in _ctypes_callproc ()
   from /home/jovyan/.conda/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so
#9  0x00007ffff681d825 in PyCFuncPtr_call ()
   from /home/jovyan/.conda/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so
#10 0x00005555556631bb in _PyObject_FastCallDict ()
#11 0x00005555556f0d3e in call_function ()
#12 0x000055555571519a in _PyEval_EvalFrameDefault ()
#13 0x00005555556ea7db in fast_function ()
#14 0x00005555556f0cc5 in call_function ()
#15 0x000055555571519a in _PyEval_EvalFrameDefault ()
#16 0x00005555556e99a6 in _PyEval_EvalCodeWithName ()
#17 0x00005555556eb108 in _PyFunction_FastCallDict ()
#18 0x000055555566339f in _PyObject_FastCallDict ()
#19 0x0000555555667ff3 in _PyObject_Call_Prepend ()
#20 0x0000555555662dde in PyObject_Call ()
#21 0x00005555556bdf6b in slot_tp_init ()
#22 0x00005555556f0f27 in type_call ()
#23 0x00005555556631bb in _PyObject_FastCallDict ()
#24 0x00005555556eacfa in _PyObject_FastCallKeywords ()
#25 0x00005555556f0d3e in call_function ()
#26 0x0000555555715eb1 in _PyEval_EvalFrameDefault ()
#27 0x00005555556eb529 in PyEval_EvalCodeEx ()
#28 0x00005555556ec2cc in PyEval_EvalCode ()
#29 0x0000555555768af4 in run_mod ()
#30 0x0000555555768ef1 in PyRun_FileExFlags ()
#31 0x00005555557690f4 in PyRun_SimpleFileExFlags ()
#32 0x000055555576cc28 in Py_Main ()
#33 0x000055555563471e in main ()

I installed hdfs3 via conda (conda install -c conda-forge hdfs3).

Here are the package versions installed by conda-forge:

    boost-cpp:    1.66.0-1         conda-forge
    bzip2:        1.0.6-1          conda-forge
    curl:         7.59.0-1         conda-forge
    hdfs3:        0.3.0-py36_0     conda-forge
    icu:          58.2-0           conda-forge
    krb5:         1.14.6-0         conda-forge
    libgcrypt:    1.8.2-hfc679d8_1 conda-forge
    libgpg-error: 1.31-hf484d3e_0  conda-forge
    libgsasl:     1.8.0-2          conda-forge
    libhdfs3:     2.3.0-2          conda-forge
    libiconv:     1.15-0           conda-forge
    libntlm:      1.4-1            conda-forge
    libprotobuf:  3.5.2-0          conda-forge
    libssh2:      1.8.0-2          conda-forge
    libuuid:      1.0.3-1          conda-forge
    libxml2:      2.9.8-0          conda-forge

I don't think there's anything wrong with the Hadoop setup, since I'm still able to access it using hadoop fs commands.

Any thoughts on what could be going wrong?

[martindurant]

libhdfs3-2.3.0-2 was just released, can you try with libhdfs3-2.3.0-1 and see if this a new problem?
conda install -c conda-forge libhdfs3=2.3.0=1

[martindurant]

@sk1p , ideas?

[lucashu1]

@martindurant that worked! Ran this in a Jupyter Notebook and things are working again: ! conda install -c conda-forge libhdfs3=2.3.0=1 hdfs3 --yes. Hopefully they're able to get that fixed before the issue pops up for more people.

[martindurant]

Thanks for testing, @lucashu1

[sk1p]

Thanks for tagging me. That's weird: effective_user=0x1 - do you have anything related to effective_user in your configuration file?

[lucashu1]

@sk1p Nope, not seeing any instances of effective_user in /etc/hadoop, which to my understanding should contain the Hadoop config files.

[martindurant]

If I remember, effective_user arises in situations something like you have logged in with a kerberos principal other than your local username. Are you using kerberos?

[sk1p]

@martindurant did anything change wrt. kerberos library dependency versions in -2?

[martindurant]

Since the last release was 9 months ago, a whole load of versions have been updated, but the latest build release was with the older code and newer versions, so that is likely not the problem. I think that there must still be bad code from the big merge ContinuumIO/libhdfs3-downstream#3 which is beyond my c++ ability. You'll notice that there were conversations around how to make that right, but it was never used in a conda build until now (the recipe still pointed to the concat branch)

[sk1p]

Ok, that may be it. Sadly I couldn't reproduce it yet, I don't have any kerberos stuff set up.

Maybe @bdrosen96 knows more?

[martindurant]

Presumably, yes, but getting this right for a variety of systems is not easy.

[eddy-geek]

I got something the same issue with a manual install, on master.

Here are the complete steps to build latest libhdfs3 master (commit 41fd50d from May 7, PR#12) on fedora 28 and reproduce this (I don't believe an actual kerberos connection is needed, just the standard hdfs-client with auth set to kerberos):

dnf install -y krb5-workstation protobuf-devel gtest-devel gmock-devel libcurl-devel libgsasl-devel libgasl
git clone https://github.com/ContinuumIO/libhdfs3-downstream
cd libhdfs3-downstream/libhdfs3
mkdir build
cd build
../bootstrap --prefix=/usr/local
make -j5
sudo make install
export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
pip3 install --user hdfs3  # 0.3.0
gdb python3
import hdfs3 ; f = hdfs3.HDFileSystem('some.net', port=8020)

[eddy-geek]

Thanks to @martindurant I was able to work around it by building the following libhdfs3 commit:

git checkout 7842951    # august
git cherry-pick 069d774 # gcc-7

(then same steps as above, it should at least get you to point where it tries to connect and fails with ERROR Failed to setup RPC connection to "some.net:8020")

[bdrosen96]

I could not reproduce this when I used a debug build. That may mean there is unitialized variable

[bdrosen96]

It is probably important to be sure the python hdfs3 code is using:

hdfsBuilderConnect.argtypes = [ct.POINTER(hdfsBuilder), ct.c_char_p]

I also could not reproduce this with non debug build. I noticed the connect call looks like:

fs = _lib.hdfsBuilderConnect(o)

instead of:

fs = _lib.hdfsBuilderConnect(o, ensure_bytes(self.effective_user))

so there might be incompatability between python lib and C++ lib