can't ssh to container

[edbond]

→  docker ps
.....
099dc3966110        hectcastro/riak:latest   /sbin/my_init --quie   8 minutes ago       Up 8 minutes        0.0.0.0:49163->8087/tcp, 0.0.0.0:49164->8098/tcp   riak01,riak02/seed,riak03/seed,riak04/seed,riak05/seed   

→  docker inspect 099dc3966110 | grep IP
        "IPAddress": "172.17.0.7",
        "IPPrefixLen": 16,
→  ssh -i insecure_key root@172.17.0.7
The authenticity of host '172.17.0.7 (172.17.0.7)' can't be established.
ECDSA key fingerprint is f1:56:8c:fc:8b:95:75:ff:5f:bc:4a:53:72:3d:da:0a.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.17.0.7' (ECDSA) to the list of known hosts.
Connection to 172.17.0.7 closed by remote host.
Connection to 172.17.0.7 closed.

[hectcastro]

Attempting to SSH with -v would be helpful to determine what went wrong. I just attempted to SSH into a container and seemed to work:

$ docker ps
CONTAINER ID        IMAGE                    COMMAND                CREATED             STATUS              PORTS                                              NAMES
0ce77a80a29c        hectcastro/riak:latest   "/sbin/my_init --qui   4 minutes ago       Up 4 minutes        0.0.0.0:49161->8087/tcp, 0.0.0.0:49162->8098/tcp   riak05
d1c442962191        hectcastro/riak:latest   "/sbin/my_init --qui   4 minutes ago       Up 4 minutes        0.0.0.0:49159->8087/tcp, 0.0.0.0:49160->8098/tcp   riak04
b0e5e154e7d5        hectcastro/riak:latest   "/sbin/my_init --qui   4 minutes ago       Up 4 minutes        0.0.0.0:49157->8087/tcp, 0.0.0.0:49158->8098/tcp   riak03
fc5a28476d5d        hectcastro/riak:latest   "/sbin/my_init --qui   4 minutes ago       Up 4 minutes        0.0.0.0:49155->8087/tcp, 0.0.0.0:49156->8098/tcp   riak02
4eac9f286dca        hectcastro/riak:latest   "/sbin/my_init --qui   4 minutes ago       Up 4 minutes        0.0.0.0:49153->8098/tcp, 0.0.0.0:49154->8087/tcp   riak01,riak02/seed,riak03/seed,riak04/seed,riak05/seed
$ docker inspect b0e5e154e7d5 | grep IPAddress
        "IPAddress": "172.17.0.10",
$ ssh -i insecure_key root@172.17.0.10
The authenticity of host '172.17.0.10 (172.17.0.10)' can't be established.
ECDSA key fingerprint is a3:0e:aa:96:22:9d:55:48:44:8f:dc:26:c0:90:b9:ba.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.17.0.10' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 12.04.4 LTS (GNU/Linux 3.8.0-35-generic x86_64)

 * Documentation:  https://help.ubuntu.com/
root@b0e5e154e7d5:~#

[edbond]

here is ssh -v output:

→  ssh -v -i insecure_key root@172.17.0.3
OpenSSH_6.4, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /home/eduard/.ssh/config
debug1: /home/eduard/.ssh/config line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 51: Applying options for *
debug1: Connecting to 172.17.0.3 [172.17.0.3] port 22.
debug1: Connection established.
debug1: identity file insecure_key type -1
debug1: identity file insecure_key-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.4
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.9p1 Debian-5ubuntu1.2
debug1: match: OpenSSH_5.9p1 Debian-5ubuntu1.2 pat OpenSSH_5*
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 zlib@openssh.com
debug1: kex: client->server aes128-ctr hmac-md5 zlib@openssh.com
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA sn:ip
debug1: Host '172.17.0.3' is known and matches the ECDSA host key.
debug1: Found key in /home/eduard/.ssh/known_hosts:1681
debug1: ssh_ecdsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering RSA public key: eduard@volcano
debug1: Authentications that can continue: publickey,password
debug1: Trying private key: insecure_key
debug1: read PEM private key done: type RSA
debug1: Enabling compression at level 6.
debug1: Authentication succeeded (publickey).
Authenticated to 172.17.0.3 ([172.17.0.3]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: Requesting authentication agent forwarding.
debug1: Sending environment.
debug1: Sending env LC_PAPER = uk_UA.utf8
debug1: Sending env LC_MONETARY = uk_UA.utf8
debug1: Sending env LC_NUMERIC = uk_UA.utf8
debug1: Sending env XMODIFIERS = @im=ibus
debug1: Sending env LANG = en_US.utf8
debug1: Sending env LC_MEASUREMENT = uk_UA.utf8
debug1: Sending env LC_TIME = uk_UA.utf8
debug1: channel 0: free: client-session, nchannels 1
Connection to 172.17.0.3 closed by remote host.
Connection to 172.17.0.3 closed.
Transferred: sent 3624, received 1472 bytes, in 0.0 seconds
Bytes per second: sent 2425815.1, received 985320.1
debug1: Exit status -1
debug1: compress outgoing: raw data 732, compressed 438, factor 0.60
debug1: compress incoming: raw data 17, compressed 10, factor 0.59

I ssh every day to remote servers without problems. Maybe locale problem?

[hectcastro]

Are you on RHEL? Some similarities in this issue regarding SELinux on RHEL: moby/moby#5032

[edbond]

yes, I'm using Fedora 20. Disabling SElinux didn't help.

snippets from ssh -vvvv

debug1: Trying private key: insecure_key
debug1: read PEM private key done: type RSA
debug3: sign_and_send_pubkey: RSA e9:5b:00:88:1f:53:ec:4b:b8:68:ce:50:f6:b4:27:66
debug2: we sent a publickey packet, wait for reply
debug1: Enabling compression at level 6.
debug1: Authentication succeeded (publickey).
Authenticated to 172.17.0.6 ([172.17.0.6]:22).

.......

debug3: Ignored env _system_name
debug3: Ignored env COLORTERM
debug3: Ignored env XAUTHORITY
debug3: Ignored env _
debug2: channel 0: request shell confirm 1
debug2: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug1: channel 0: free: client-session, nchannels 1
debug3: channel 0: status: The following connections are open:
  #0 client-session (t4 r0 i0/0 o0/0 fd 4/5 cc -1)

Connection to 172.17.0.6 closed by remote host.
Connection to 172.17.0.6 closed.
Transferred: sent 3624, received 1472 bytes, in 0.0 seconds
Bytes per second: sent 1473740.3, received 598605.3
debug1: Exit status -1
debug1: compress outgoing: raw data 732, compressed 437, factor 0.60
debug1: compress incoming: raw data 17, compressed 10, factor 0.59

[hectcastro]

Can you please provide the output of docker version? I'll try to reproduce on Fedora 20.

[edbond]

→  docker version
Client version: 1.1.2
Client API version: 1.13
Go version (client): go1.2.2
Git commit (client): d84a070/1.1.2
Server version: 1.1.2
Server API version: 1.13
Go version (server): go1.2.2
Git commit (server): d84a070/1.1.2

Thank you

[edbond]

→  yum info docker-io
Loaded plugins: langpacks, refresh-packagekit
Installed Packages
Name        : docker-io
Arch        : x86_64
Version     : 1.1.2
Release     : 2.fc20

[hectcastro]

I just tried reproducing on a Fedora 20 Vagrant box. The main modification, other than installing the docker-io package, was to edit /etc/sysconfig/docker:

[vagrant@localhost ~]$ cat /etc/sysconfig/docker
# /etc/sysconfig/docker
OPTIONS=--selinux-enabled -H tcp://127.0.0.1:2375

From there, I downloaded the insecure_key and set the DOCKER_HOST environmental variable:

[vagrant@localhost ~]$ export DOCKER_HOST="tcp://127.0.0.1:2375"

Creating the Riak cluster succeeded:

[vagrant@localhost vagrant]$ DOCKER_RIAK_AUTOMATIC_CLUSTERING=1 DOCKER_RIAK_CLUSTER_SIZE=5 make start-cluster
./bin/start-cluster.sh

Bringing up cluster nodes:

  Successfully brought up [riak01]
  Successfully brought up [riak02]
  Successfully brought up [riak03]
  Successfully brought up [riak04]
  Successfully brought up [riak05]

Please wait approximately 30 seconds for the cluster to stabilize.

Then, I got the internal IP of one of the containers:

[vagrant@localhost vagrant]$ docker inspect a4737230156a | grep IPAddress
        "IPAddress": "172.17.0.4",

And successfully connected via SSH:

[vagrant@localhost ~]$ ssh -vvvv -i insecure_key root@172.17.0.4
OpenSSH_6.4, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 51: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to 172.17.0.4 [172.17.0.4] port 22.
debug1: Connection established.
debug3: Incorrect RSA1 identifier
debug3: Could not load "insecure_key" as a RSA1 public key
debug1: identity file insecure_key type -1
debug1: identity file insecure_key-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.4
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.9p1 Debian-5ubuntu1.2
debug1: match: OpenSSH_5.9p1 Debian-5ubuntu1.2 pat OpenSSH_5*
debug2: fd 3 setting O_NONBLOCK
debug3: load_hostkeys: loading entries for host "172.17.0.4" from file "/home/vagrant/.ssh/known_hosts"
debug3: load_hostkeys: found key type ECDSA in file /home/vagrant/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-256-96,hmac-sha2-512,hmac-sha2-512-96,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-256-96,hmac-sha2-512,hmac-sha2-512-96,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: found hmac-md5
debug1: kex: server->client aes128-ctr hmac-md5 none
debug2: mac_setup: found hmac-md5
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA 1f:d6:c5:e4:1c:33:cd:5e:62:66:8a:2c:ab:3b:9f:12
debug3: load_hostkeys: loading entries for host "172.17.0.4" from file "/home/vagrant/.ssh/known_hosts"
debug3: load_hostkeys: found key type ECDSA in file /home/vagrant/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys
debug1: Host '172.17.0.4' is known and matches the ECDSA host key.
debug1: Found key in /home/vagrant/.ssh/known_hosts:1
debug1: ssh_ecdsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: insecure_key ((nil)), explicit
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: insecure_key
debug1: read PEM private key done: type RSA
debug3: sign_and_send_pubkey: RSA e9:5b:00:88:1f:53:ec:4b:b8:68:ce:50:f6:b4:27:66
debug2: we sent a publickey packet, wait for reply
debug1: Authentication succeeded (publickey).
Authenticated to 172.17.0.4 ([172.17.0.4]:22).
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug2: callback start
debug2: fd 3 setting TCP_NODELAY
debug3: packet_set_tos: set IP_TOS 0x10
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug1: Sending environment.
debug3: Ignored env XDG_SESSION_ID
debug3: Ignored env HOSTNAME
debug3: Ignored env TERM
debug3: Ignored env SHELL
debug3: Ignored env HISTSIZE
debug3: Ignored env SSH_CLIENT
debug3: Ignored env DOCKER_HOST
debug3: Ignored env OLDPWD
debug3: Ignored env SSH_TTY
debug3: Ignored env USER
debug3: Ignored env LS_COLORS
debug3: Ignored env MAIL
debug3: Ignored env PATH
debug3: Ignored env PWD
debug1: Sending env LANG = en_US.UTF-8
debug2: channel 0: request env confirm 0
debug3: Ignored env HISTCONTROL
debug3: Ignored env SHLVL
debug3: Ignored env HOME
debug3: Ignored env LOGNAME
debug3: Ignored env SSH_CONNECTION
debug3: Ignored env LESSOPEN
debug3: Ignored env XDG_RUNTIME_DIR
debug3: Ignored env _
debug2: channel 0: request shell confirm 1
debug2: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug2: channel_input_status_confirm: type 99 id 0
debug2: PTY allocation request accepted on channel 0
debug2: channel 0: rcvd adjust 2097152
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0
Last login: Mon Sep  1 17:06:04 2014 from 172.17.42.1
Welcome to Ubuntu 12.04.4 LTS (GNU/Linux 3.8.0-35-generic x86_64)

 * Documentation:  https://help.ubuntu.com/
root@a4737230156a:~#

Any noticeable differences between those steps and what you're doing?

[edbond]

I run the same commands, here is a patch comparing ssh -vvvv output: https://gist.github.com/edbond/fdaf0f82dbec799f3876

[hectcastro]

You may want to try using this method of running commands within a Docker container. Perhaps then, you could restart SSH on another port without daemonizing to see if that provides any more information.

[cosmin-marginean]

Having similar issues on OSX

$ ssh -vvvv -i insecure_key root@172.17.0.2
OpenSSH_6.9p1, LibreSSL 2.1.8
debug1: Reading configuration data /Users/cosmin/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to 172.17.0.2 [172.17.0.2] port 22.

Stuck here ^

Cluster seems alright though:

$ make test-cluster | egrep -A6 "ring_members"
    "ring_members": [
        "riak@172.17.0.2",
        "riak@172.17.0.3",
        "riak@172.17.0.4",
        "riak@172.17.0.5",
        "riak@172.17.0.6"
    ],

It's essential for us to ssh though as we need to perform some riak-admin tasks. Any suggestions are welcome.