Skip to content

Remote Shell With Caravan

Chris Brodt edited this page Dec 18, 2017 · 1 revision

For most cases, connecting to remote shells will actually be easier with Caravan than when using epmd. There's no need to create SSH tunnels to coordinate the epmd port negotiation, you just have to start your iex session with the Caravan modules and a --name option with an open port.

This can become tricky however, if you're using a custom DNS server that isn't already part of your system's resolve.conf file (or whatever). So, if the nodes running in your production/staging/whatever environment are configured with a --name that's only resolvable via a specific DNS server (eg. Consul DNS interface), you'll need to make sure your local IEx session can talk to that DNS server.

Erlang, as in most cases, does have a solution for you. There is a config file that can be used to add DNS servers and tell the Erlang VM what resolution methods to use. What was not obvious, is that I was unable to use the nameserver config parameter to add a nameserver. It only worked when I used the alt_nameserver parameter. If anyone out there with more Erlang knowledge can share why that's the case, I would love to hear from you. Anyways, here's how I configured it to work.

Step 1: Create an inet_cfg file

I put this in the config directory of my mix project. This file is basically just Erlang code, so mind the . at the end of each line.

%% -- ERLANG INET CONFIGURATION FILE --
{alt_nameserver, {10,0,254,75}, 8600}.
{lookup, [file, dns]}.

In my example, I have a DNS server using a non standard port (8600). If your DNS is going to be the standard port 53, than you can leave that part off.

Step 2: Add command line flag to IEx

You'll need to add the following flag to IEx:--erl "-kernel inetrc './config/erl_inetrc'". This needs to be in addition to the existing flags required to use Caravan. Here's a complete example:

iex --erl "-pa ../../_build/dev/lib/caravan/ebin -proto_dist Elixir.Caravan.Epmd.Dist" \
--erl "-start_epmd false -epmd_module Elixir.Caravan.Epmd.Client"  \
--erl "-kernel inetrc './config/erl_inetrc'" --name "$1" --cookie 'foocookie' -S mix

Step 3: Start IEx and verify config

Once in IEx running :inet.get_rc() should return something similar to this:

[{:nameservers, {10, 0, 0, 2}}, {:nameservers, {8, 8, 8, 8}},
 {:nameservers, {192, 168, 1, 1}}, {:alt_nameservers, {10, 0, 254, 75}, 8600},
 {:resolv_conf, '/etc/resolv.conf'}, {:hosts_file, '/etc/hosts'},
 {:lookup, [:file, :dns]}]

The important thing to check is that alt_nameservers is present and the correct value. Elixir will complain if the config file isn't valid on startup, so make sure to watch for that.

Step 4: Connect to your remote node and do fun remote shell things

Run Node.ping(somenode-23484@somewhere.net.consul), substituting your actual remote node name, and you should get a response of :pong on success. Now you can run Observer or use the :rpc module to execute functions on the remote node and see the output locally.

That didn't work! I am so mad right now

I hear you, but lets try a couple of things to see what the issue could be.

Verify that the name is resolvable via the DNS server you think it is

You can use Erlang's built-in DNS client and pass it the DNS server directly:

:inet_res.lookup(to_charlist("somewhere.net.consul"), :in, :a, [nameservers: [{{10,0,252,160}, 8600}]])

IMPORTANT: Remember that we're only looking up in DNS the part of the remote node --name AFTER the @. A correct result will be a list with IP tuples. If you get an :nxdomain double check the name you queried, DNS server IP, or the DNS server itself via dig.

Checking the DNS client is configured correctly

If the above works, you know you have the correct DNS server. Now check to see if :inet_res will use it by default.

:inet_res.getbyname(to_charlist("somewhere.net.consul"), :a)

As above, you should get something with an IP tuple or an error. If it's an error, that it's something wrong with your inet config, or some other issue with your setup. Check the network, etc.

Both of the above checked out, but it still doesn't work. I'm SUPER mad now

If both of the above commands worked, the issue is likely with a firewall or node configuration. Double check the --name parameters, make sure all nodes are setup to use Caravan for node distribution, etc.