resolve_private_address() fails

[chrisheckler]

If your DNS can't resolve hostnames of other machines on your network this will fail, in which causing charm install/deploys to fail.

	unit-spark-1: 15:57:21 ERROR unit.spark/1.juju-log zookeeper:7: Unable to resolve private IP: juju-e32cb2-3.hecky (will attempt to guess)
unit-spark-1: 15:57:21 ERROR unit.spark/1.juju-log zookeeper:7: [Errno -2] Name or service not known
unit-spark-1: 15:57:21 DEBUG unit.spark/1.zookeeper-relation-changed Traceback (most recent call last):
unit-spark-1: 15:57:21 DEBUG unit.spark/1.zookeeper-relation-changed   File "/usr/local/lib/python3.5/dist-packages/jujubigdata/utils.py", line 433, in resolve_private_address
unit-spark-1: 15:57:21 DEBUG unit.spark/1.zookeeper-relation-changed     ip = socket.gethostbyname(addr)
unit-spark-1: 15:57:21 DEBUG unit.spark/1.zookeeper-relation-changed socket.gaierror: [Errno -2] Name or service not known
unit-spark-1: 15:57:21 DEBUG unit.spark/1.zookeeper-relation-changed 
unit-spark-1: 15:57:21 DEBUG unit.spark/1.zookeeper-relation-changed During handling of the above exception, another exception occurred:
unit-spark-1: 15:57:21 DEBUG unit.spark/1.zookeeper-relation-changed 
unit-spark-1: 15:57:21 DEBUG unit.spark/1.zookeeper-relation-changed Traceback (most recent call last):
unit-spark-1: 15:57:21 DEBUG unit.spark/1.zookeeper-relation-changed   File "/var/lib/juju/agents/unit-spark-1/charm/hooks/zookeeper-relation-changed", line 19, in <module>
unit-spark-1: 15:57:21 DEBUG unit.spark/1.zookeeper-relation-changed     main()
unit-spark-1: 15:57:21 DEBUG unit.spark/1.zookeeper-relation-changed   File "/usr/local/lib/python3.5/dist-packages/charms/reactive/__init__.py", line 78, in main
unit-spark-1: 15:57:21 DEBUG unit.spark/1.zookeeper-relation-changed     bus.dispatch()
unit-spark-1: 15:57:21 DEBUG unit.spark/1.zookeeper-relation-changed   File "/usr/local/lib/python3.5/dist-packages/charms/reactive/bus.py", line 423, in dispatch
unit-spark-1: 15:57:21 DEBUG unit.spark/1.zookeeper-relation-changed     _invoke(other_handlers)
unit-spark-1: 15:57:21 DEBUG unit.spark/1.zookeeper-relation-changed   File "/usr/local/lib/python3.5/dist-packages/charms/reactive/bus.py", line 406, in _invoke
unit-spark-1: 15:57:21 DEBUG unit.spark/1.zookeeper-relation-changed     handler.invoke()
unit-spark-1: 15:57:21 DEBUG unit.spark/1.zookeeper-relation-changed   File "/usr/local/lib/python3.5/dist-packages/charms/reactive/bus.py", line 280, in invoke
unit-spark-1: 15:57:21 DEBUG unit.spark/1.zookeeper-relation-changed     self._action(*args)
unit-spark-1: 15:57:21 DEBUG unit.spark/1.zookeeper-relation-changed   File "/var/lib/juju/agents/unit-spark-1/charm/reactive/spark.py", line 203, in reinstall_spark
unit-spark-1: 15:57:21 DEBUG unit.spark/1.zookeeper-relation-changed     install_spark_standalone(zks, peers)
unit-spark-1: 15:57:21 DEBUG unit.spark/1.zookeeper-relation-changed   File "/var/lib/juju/agents/unit-spark-1/charm/reactive/spark.py", line 99, in install_spark_standalone
unit-spark-1: 15:57:21 DEBUG unit.spark/1.zookeeper-relation-changed     spark.configure(hosts, zk_units=zks, peers=peers, extra_libs=extra_libs)
unit-spark-1: 15:57:21 DEBUG unit.spark/1.zookeeper-relation-changed   File "lib/charms/layer/bigtop_spark.py", line 234, in configure
unit-spark-1: 15:57:21 DEBUG unit.spark/1.zookeeper-relation-changed     master_ip = utils.resolve_private_address(available_hosts['spark-master'])
unit-spark-1: 15:57:21 DEBUG unit.spark/1.zookeeper-relation-changed   File "/usr/local/lib/python3.5/dist-packages/jujubigdata/utils.py", line 440, in resolve_private_address
unit-spark-1: 15:57:21 DEBUG unit.spark/1.zookeeper-relation-changed     raise ValueError('Unable to resolve or guess IP from private-address: %s' % addr)
unit-spark-1: 15:57:21 DEBUG unit.spark/1.zookeeper-relation-changed ValueError: Unable to resolve or guess IP from private-address: juju-e32cb2-3.hecky
unit-spark-1: 15:57:21 ERROR juju.worker.uniter.operation hook "zookeeper-relation-changed" failed: exit status 1

Can we sort this out a different way please?

Thanks

[jamesbeedy]

I can verify this same behavior happens in scenarios where there is no node <-> node fqdn resolution.

Example:

If a user assigns the lxd bridge to the primary interface of their host to get the the lxd onto the LAN, usually the dhcp/dns server which is the modem/router will provide a basic search domain such that lxd instances will get fqdn of juju-5rhsu4-1.<router-or-modem-search-domain>. While the router/modem may provide the local search domain and possibly basic dns forwarding, it is not tracking and resolving dns for each device on the network. Thus if the charm code tries to resolve another service by hostname or fqdn it will fail.

[jamesbeedy]

I was able to resolve this by using MAAS to provide dns/dhcp on the local network such that the lxd could resolve eachother.

[chrisheckler]

@jamesbeedy this solved my issue too! thanks!

not sure it fixes the bug though ..

[kwmonroe]

@chrisheckler it for sure does not fix this bug, but I do appreciate a good workaround - @jamesbeedy good to hear maas dns (or any real dns) solves the problem :)

The underlying issue is that most big data components really really want to be able to resolve IPs, and so the charms really really want that too. There are reasons for that, for example the "false" warning for things like dfs.namenode.datanode.registration.ip-hostname-check here:

https://hadoop.apache.org/docs/r2.7.3/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml

For that particular limitation, namenode and datanode charms will check for dns resolution, and when not available, they'll set the above to false and try their best to make an hdfs cluster. Other scenarios don't have the same mitigations, so charms will outright fail if proper dns isn't available.

In the past, we've tried maintaining our own copy of /etc/hosts and distributing that around the cluster as needed (see all the etc_hosts methods like this one). It was hacky, but it worked well in the cases when proper dns wasn't available. The problems came when trying to maintain /etc/hosts in envs that do have proper dns. Inevitably, somewhere down in puppet, you'll fall out to facter(fqdn) and that will take the real dns hostname, which won't match our contrived /etc/hosts entry, and all hell will break loose.

Ok, maybe not all hell, but it'll fail, and it'll suck. I'm sure we could do better with our /etc/hosts methods, and get them in a place where that behavior would work well in all environments. For that reason, I'll leave this issue open for now. We can do better, but the current workaround is to make proper dns services available.