LND HA guide : question about bitcoind resliency

[AirHache]

Hi,

I'm currently following the LND Ha Guide, and I am wondering about Bitcoind resiliency.

I understood that Bitcoind would run on 1 server, while LND would be run on 3 separate nodes.
However, if the Bitcoind server crashes, how can we ensure that LND still works, without Bitcoin data ?

[Filiprogrammer]

Hi!

You're correct that in the described setup bitcoind is a single point of failure.
If bitcoind goes down, LND will continue running for a short while but will not confirm new channel states. Eventually the chain backend health check will fail and the active LND instance will shut down.

However, as long as the remote watchtower is running, there is at least no risk of losing funds.

To keep the LND cluster running in such a case, redundancy for bitcoind would also need to be set up:

1. The simplest and most robust solution would be to run an instance of bitcoind alongside LND on each of the 3 hosts and point each LND to the local RPC and ZMQ endpoints. The drawback of this is the additional storage cost, since 3 copies of the Blockchain would have to be stored.
2. Another approach would be 2 or more seperate hosts running bitcoind with failover behind a floating IP address. LND nodes would all point to that single floating IP address.

I should probably update the setups in the guide to also be resilient against bitcoind failure.

[AirHache]

Hi,
Thank you for your answer.

Now i am wondering something else. The architecture displayed are showing the 3 etcd/patroni/lnd nodes in the same datacenter.

Do we agree it would be advised to locate them in another places, or at least plugged on another internet connection?

[Filiprogrammer]

Putting the nodes in different datacenters isn’t usually recommended unless you have very fast, low-latency links between them. Latency directly impacts performance, since every write to the PostgreSQL database needs to be replicated. So if this is supposed to work with reasonably good performance, the datacenters should not be too far apart.

A simpler way to improve resilience is to build a fully-meshed internal cluster network: Each node has a direct Ethernet connection to every other node, without a router or switch that can fail in-between. Internet access should be on another network interface separate from cluster communication.

And sure, if you want to go all the way, each node can indeed have its own independent internet connection. A local monitoring service could be set up on each node to trigger failover if one uplink goes down.

[AirHache]

Thank you for your answers.
Very clear.