degraded cluster

[nickyruggs]

I am getting these errors in gostats.log and consequently no information is visible on the dashboards.

24-04-11T13:29:02-07:00 influxdb.go:72 WARNING Unable to retrieve stat cluster.cpu.intr.avg from cluster pscale-nvda, error Degraded result, some node input missing
2024-04-11T13:29:02-07:00 influxdb.go:72 WARNING Unable to retrieve stat cluster.cpu.user.avg from cluster pscale-nvda, error Degraded result, some node input missing
2024-04-11T13:29:02-07:00 influxdb.go:72 WARNING Unable to retrieve stat cluster.cpu.sys.avg from cluster pscale-nvda, error Degraded result, some node input missing
2024-04-11T13:29:02-07:00 influxdb.go:72 WARNING Unable to retrieve stat node.cpu.throttling from cluster pscale-nvda, error Key does not exist with current configuration
2024-04-11T13:29:02-07:00 influxdb.go:72 WARNING Unable to retrieve stat cluster.net.ext.bytes.in.rate from cluster pscale-nvda, error Degraded result, some node input missing

Is it possible to allow for degraded clusters? in general most clusters would be fixed quickly but having a gap is not ideal. if it was a switch in the idic.toml then users could use it during these times and go back to full mode later if it made a difference.

[tenortim]

The collector already passes '?degraded=true' which should pull information. I think the endpoint is both returning an error and maybe some data too in which case, I'll need to add a special case for this error type to have it treat is "not an error". Need to set up a test to confirm.

[fhutchinson-ES]

We have the same errors on our cluster. Nodes are under going firmware upgrades so the cluster is technically degraded. We then lose all our metrics because one node is down at a time when we are particularly interested in them to check the status of the cluster.