How many clusters the daemon can manage

[ut0f9t]

The daemon is not able to start when the number of clusters is superior at 8.
How can i manage that by using 2 deamons and 2 configuratin files?

[lukeskaivolker]

Strange, as I have used to monitor over 60 clusters with one daemon running.
What error you get when starting daemon? How about server performance on which you are running daemon?

[ut0f9t]

Thanks for your confirmation... its look like one cluster return an error and the daemon hang (config have to be checked)

2019-10-16 09:18:31,920:urllib3.connectionpool:WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fa92
b7f06a0>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /platform/3/cluster/config

but wit the current configuration after removed the cluster on error we can see this on the log
2019-10-16 09:26:18,330 WARNING Connection pool is full, discarding connection:

[ut0f9t]

2019-10-16 09:28:57,264:urllib3.connectionpool:WARNING: Connection pool is full, discarding connection

[tenortim]

This is a concern with the way the collector schedules collection from multiple clusters (one problematic cluster can have knock-on effects). I have rewritten the collector in Go and plan to push the new version to github shortly. I intend to update the Python collector to Python 3 and to leverage the async functionality, but currently, the Go collector is more reliable and better-performing.

[ut0f9t]

Thks .. look forward to the new version in Go bye thks again

[tenortim]

I still haven't posted the Go collector externally (I will, I promise), but as noted above, the Python collector is certainly not limited to 8 clusters (we're using internally for close to 20). It is rather fragile wrt errors at startup however, and I need to fix that. In this case, the startup error is because it can't resolve the cluster name (but the urllib3 errors are terrible and don't actually tell you the name part of the URL). The connectionpool warnings are annoying (I opened a bug for those), but harmless in that they do not prevent the collector working (it just means that there are more connections than the cached limited so the collector is having to reconnect more frequently than we would like).

[ut0f9t]

Hello Tim
Many thanks .. I was thinking about a problem with the connection pool with python 2, you gave a python3 version of the collector that I never tried. I'll keep you informed.
Thks again

[tenortim]

FYI, the Golang collector is available at https://github.com/tenortim/gostats