Kubernetes Cluster Quota Predictions Predictions are wrong - clusters being made that can never create

[AlexCK-STFC]

Quota predictions are wrong, especially for non-standard control plane counts.

This is bad, as its allowing clusters to create that go over quota, and CAPO gets stuck 409-ing forever.

See:

azimuth/api/azimuth/cluster_api/base.py

Lines 128 to 154 in e70f579

	def _from_api_cluster_template(self, ct):
	"""
	Converts a cluster template from the Kubernetes API to a DTO.
	"""
	values = ct.spec["values"]
	# We only need to account for the etcd volume if it has type Volume
	etcd_volume_size = 0
	etcd_volume = values.get("etcd", {}).get("blockDevice")
	if etcd_volume and etcd_volume.get("type", "Volume") == "Volume":
	etcd_volume_size = etcd_volume["size"]
	return dto.ClusterTemplate(
	ct.metadata.name,
	ct.spec.label,
	ct.spec.get("description"),
	values["kubernetesVersion"],
	ct.spec.get("deprecated", False),
	values.get("controlPlane", {}).get("machineCount", 3),
	etcd_volume_size,
	values.get("controlPlane", {}).get("machineRootVolume", {}).get("diskSize")
	or 0,
	values.get("nodeGroupDefaults", {})
	.get("machineRootVolume", {})
	.get("diskSize")
	or 0,
	ct.spec.get("tags", []),
	dateutil.parser.parse(ct.metadata["creationTimestamp"]),
	)

If I'm tracing the logic right, it's using values like controlPlane.machineCount, and root volume disk size, that don't actually exist in the cluster templates:
https://github.com/azimuth-cloud/ansible-collection-azimuth-ops/blob/e70cc3d808af3b206033d04c595860aeb74660b3/roles/azimuth_capi_operator/defaults/main.yml#L557-L566

I.e. consider the quota predictions for this cluster, which has 6 nodes (5 control plane + 1 worker) each with 2 CPU, 8GB RAM, 50GB disk:

• CPUs is wrong, it's assuming 8 more cores -> only 4 new machines
  • Because, as linked above, machineCount is missing and defaults to 3. Then there's 1 worker node.
• RAM is wrong, it's assuming 32GB more -> 4 new machines
  • Same as CPU
• Volume Storage is potentially wrong, its only counting the 3*10GB from Metrics/Logs/Alertmanager, not any control plane/node group volumes. Here, though, we haven't set root volume specs for the machines so it's just using the ephemeral root disk for the flavour.
  • Similar to above, rootVolume disk size is missing and defaults to 0, diskSize is missing and defaults to 0. The Metrics/Logs/Alertmanager volumes total the 30GB
• Machine count is somehow right here. But on Dev we use 1 control plane node, then it is wrong.

I'm struggling to trace all the value sources as I'm unfamiliar with the codebase. Apologies for this issue being confusing. But I hope I've conveyed the central issue.

But as I understand it:

• Some values are properly found from the node sizes/flavours:
  

  azimuth/api/azimuth/views.py

  Lines 1351 to 1399 in e70f579

  	def kubernetes_cluster_check_quotas(session, cluster, template, **data):
  	"""
  	Check the quotas for a Kubernetes cluster.
  	"""
  	calculator = scheduling.KubernetesClusterCalculator(session)
  	# Calculate the resources used by the current cluster
  	if cluster:
  	# Index the sizes that have already been loaded so we don't have to load them
  	# again
  	known_sizes = {}
  	if "control_plane_size" in data:
  	known_sizes[data["control_plane_size"].id] = data["control_plane_size"]
  	for ng in data.get("node_groups", []):
  	known_sizes[ng["machine_size"].id] = ng["machine_size"]
  	# Calculate the data for the current state of the cluster
  	current_data = {
  	"control_plane_size": (
  	known_sizes[cluster.control_plane_size_id]
  	if cluster.control_plane_size_id in known_sizes
  	else session.find_size(cluster.control_plane_size_id)
  	),
  	"node_groups": [
  	{
  	"name": ng.name,
  	"machine_size": (
  	known_sizes[ng.machine_size_id]
  	if ng.machine_size_id in known_sizes
  	else session.find_size(ng.machine_size_id)
  	),
  	"autoscale": ng.autoscale,
  	"count": ng.count,
  	"min_count": ng.min_count,
  	"max_count": ng.max_count,
  	}
  	for ng in cluster.node_groups
  	],
  	"monitoring_enabled": cluster.monitoring_enabled,
  	"monitoring_metrics_volume_size": cluster.monitoring_metrics_volume_size,
  	"monitoring_logs_volume_size": cluster.monitoring_logs_volume_size,
  	}
  	# Calculate the resources for the current state of the cluster
  	current_resources = calculator.calculate(template, **current_data)
  	# Overwrite with any changes from the incoming data
  	data = {**current_data, **data}
  	else:
  	current_resources = None
  	future_resources = calculator.calculate(template, **data)
  	checker = scheduling.QuotaChecker(session)
  	return [future_resources, *checker.check(future_resources, current_resources)]

  
  

  azimuth/api/azimuth/views.py

  Lines 1426 to 1428 in e70f579

  	_, fits, quotas = kubernetes_cluster_check_quotas(
  	session, None, **input_serializer.validated_data
  	)

• Other values, like control plane count and node volume information, are being found from the template but don't exist there.
  

  azimuth/api/azimuth/cluster_api/base.py

  Lines 128 to 154 in e70f579

  	def _from_api_cluster_template(self, ct):
  	"""
  	Converts a cluster template from the Kubernetes API to a DTO.
  	"""
  	values = ct.spec["values"]
  	# We only need to account for the etcd volume if it has type Volume
  	etcd_volume_size = 0
  	etcd_volume = values.get("etcd", {}).get("blockDevice")
  	if etcd_volume and etcd_volume.get("type", "Volume") == "Volume":
  	etcd_volume_size = etcd_volume["size"]
  	return dto.ClusterTemplate(
  	ct.metadata.name,
  	ct.spec.label,
  	ct.spec.get("description"),
  	values["kubernetesVersion"],
  	ct.spec.get("deprecated", False),
  	values.get("controlPlane", {}).get("machineCount", 3),
  	etcd_volume_size,
  	values.get("controlPlane", {}).get("machineRootVolume", {}).get("diskSize")
  	or 0,
  	values.get("nodeGroupDefaults", {})
  	.get("machineRootVolume", {})
  	.get("diskSize")
  	or 0,
  	ct.spec.get("tags", []),
  	dateutil.parser.parse(ct.metadata["creationTimestamp"]),
  	)

  • These should be being added to the template in azimuth-ops, or found dynamically like the other values.

Side Note: It'd be nice to have quota checks for additional OpenStack quotas that aren't already here, like network security group count. These should also be added to the "Quotas" page on the sidebar.

[sd109]

Thanks for raising this @AlexCK-STFC - a few questions / comments:

• Machine count is somehow right here. But on Dev we use 1 control plane node, then it is wrong.

I think machine count is actually wrong in the same way as the others here? It's showing a change from 2 -> 6 nodes when it should be 2 -> 8 nodes, right?

• Volume Storage is potentially wrong, its only counting the 3*10GB from Metrics/Logs/Alertmanager, not any control plane/node group volumes. Here, though, we haven't set root volume specs for the machines so it's just using the ephemeral root disk for the flavour.
  • Similar to above, rootVolume disk size is missing and defaults to 0, diskSize is missing and defaults to 0. The Metrics/Logs/Alertmanager volumes total the 30GB

I'm not sure I understand this part. If your clusters are not configured to use Cinder volume-backed root disks then it will just use standard Nova-backed root disks which are not part of the OpenStack volumes service and therefore do not count towards volumes quota. In which case, the only volumes that need to be accounted for are any PVCs which use the Cinder CSI storage class (since they are backed by Cinder volumes) and in a standard cluster there are only 3 such PVCs which are the metrics, logs and alertmanager volumes that you mention. Am I missing something?

To summarise, I think the only(?) issue here is that we're just assuming control plane count is always 3 here in quota calculations? If so, that should be an easy fix and we'll take a look at fixing that ASAP.

[sd109]

Having looked into this some more, the general problem is that quota calculations are not accounting for any value overrides that are passed directly from the Azimuth CAPI operator to CAPI Helm charts e.g. here: https://github.com/stfc/cloud-azimuth-config/blob/fb98e5e04a73a28be5be399cfd06f33ac524af12/environments/stfc-base/inventory/group_vars/all/variables.yml#L124

I'll have a think about how best to solve this and we'll let you know when a fix is available.

[AlexCK-STFC]

Having looked into this some more, the general problem is that quota calculations are not accounting for any value overrides that are passed directly from the Azimuth CAPI operator to CAPI Helm charts e.g. here: https://github.com/stfc/cloud-azimuth-config/blob/fb98e5e04a73a28be5be399cfd06f33ac524af12/environments/stfc-base/inventory/group_vars/all/variables.yml#L124

I'll have a think about how best to solve this and we'll let you know when a fix is available.

There's that, but also that it's relying on controlPlane.machineCount inside the CAPI template values which doesn't exist. Then falls back to 3, which has been a fine guess until we changed it. Unless I'm misunderstanding the template loading.

azimuth/api/azimuth/cluster_api/base.py

Lines 128 to 186 in e70f579

	def _from_api_cluster_template(self, ct):
	"""
	Converts a cluster template from the Kubernetes API to a DTO.
	"""
	values = ct.spec["values"]
	# We only need to account for the etcd volume if it has type Volume
	etcd_volume_size = 0
	etcd_volume = values.get("etcd", {}).get("blockDevice")
	if etcd_volume and etcd_volume.get("type", "Volume") == "Volume":
	etcd_volume_size = etcd_volume["size"]
	return dto.ClusterTemplate(
	ct.metadata.name,
	ct.spec.label,
	ct.spec.get("description"),
	values["kubernetesVersion"],
	ct.spec.get("deprecated", False),
	values.get("controlPlane", {}).get("machineCount", 3),
	etcd_volume_size,
	values.get("controlPlane", {}).get("machineRootVolume", {}).get("diskSize")
	or 0,
	values.get("nodeGroupDefaults", {})
	.get("machineRootVolume", {})
	.get("diskSize")
	or 0,
	ct.spec.get("tags", []),
	dateutil.parser.parse(ct.metadata["creationTimestamp"]),
	)
	@convert_exceptions
	def cluster_templates(self) -> t.Iterable[dto.ClusterTemplate]:
	"""
	Lists the cluster templates currently available to the tenancy.
	"""
	self._log("Fetching available cluster templates")
	templates = list(
	self._client.api(AZIMUTH_API_VERSION).resource("clustertemplates").list()
	)
	# Filter cluster templates based on ACL annotations
	tenancy = self._cloud_session.tenancy()
	templates = [t for t in templates if allowed_by_acls(t, tenancy)]
	self._log("Found %s cluster templates", len(templates))
	return tuple(self._from_api_cluster_template(ct) for ct in templates)
	@convert_exceptions
	def find_cluster_template(self, id: str) -> dto.ClusterTemplate: # noqa: A002
	"""
	Finds a cluster template by id.
	"""
	self._log("Fetching cluster template with id '%s'", id)
	template = (
	self._client.api(AZIMUTH_API_VERSION).resource("clustertemplates").fetch(id)
	)
	if not allowed_by_acls(template, self._cloud_session.tenancy()):
	raise errors.ObjectNotFoundError(f"Cannot find cluster template {id}")
	return self._from_api_cluster_template(template)

For controlPlane.machinecount, and the root volume sizes etc, seems like its loading from the clustertemplates kubernetes resource; which is defined like so:
https://github.com/azimuth-cloud/ansible-collection-azimuth-ops/blob/e70cc3d808af3b206033d04c595860aeb74660b3/roles/azimuth_capi_operator/tasks/main.yml#L15-L37

And doesn't actually contain any of those values. At a minimum, the cluster templates should contain these. But ideally it'd load them dynamically like the rest of the values, as you say ideally it'd take into account value overrides to the helm charts.

[AlexCK-STFC]

Thanks for raising this @AlexCK-STFC - a few questions / comments:

• Machine count is somehow right here. But on Dev we use 1 control plane node, then it is wrong.

I think machine count is actually wrong in the same way as the others here? It's showing a change from 2 -> 6 nodes when it should be 2 -> 8 nodes, right?

• Volume Storage is potentially wrong, its only counting the 3*10GB from Metrics/Logs/Alertmanager, not any control plane/node group volumes. Here, though, we haven't set root volume specs for the machines so it's just using the ephemeral root disk for the flavour.

  • Similar to above, rootVolume disk size is missing and defaults to 0, diskSize is missing and defaults to 0. The Metrics/Logs/Alertmanager volumes total the 30GB

I'm not sure I understand this part. If your clusters are not configured to use Cinder volume-backed root disks then it will just use standard Nova-backed root disks which are not part of the OpenStack volumes service and therefore do not count towards volumes quota. In which case, the only volumes that need to be accounted for are any PVCs which use the Cinder CSI storage class (since they are backed by Cinder volumes) and in a standard cluster there are only 3 such PVCs which are the metrics, logs and alertmanager volumes that you mention. Am I missing something?

To summarise, I think the only(?) issue here is that we're just assuming control plane count is always 3 here in quota calculations? If so, that should be an easy fix and we'll take a look at fixing that ASAP.

Sorry I hadn't read this yet.

What I meant by volume storage is potentially wrong is yes, here we're just using nova root disks which don't count to quota so it's fine.

But there's potentially a bug for people who do use Cinder volume backed root disks; for the same reason machine count is wrong, as it's trying to grab details about them from the Kubernetes ClusterTemplate resource/the CAPI template which don't exist.