Add ACNS observability test with Cilium dataplane by carlotaarvela · Pull Request #1053 · Azure/telescope

carlotaarvela · 2026-02-11T17:11:22Z

Summary

This PR adds a Container Network Logs (CNL) performance evaluation scenario for AKS + Azure CNI Overlay + Cilium dataplane. It includes the ClusterLoader2 (CL2) workload, Prometheus monitors, and measurement queries needed to evaluate CNL observability overhead at scale.

The scenario is designed to validate Container Network Logs performance by:

Provisioning a 1000-node AKS cluster with Kubernetes 1.33+ (required for CNL)
Deploying Fortio server/client pairs to generate sustained in-cluster traffic
Creating ContainerNetworkLog CRDs to enable flow log collection
Measuring the overhead of CNL components (AMA-logs, Cilium/Hubble) on CPU, memory, and disk I/O

What's measured

Component	Metrics
AMA-Logs agent	CPU/memory usage, FS write throughput/latency, container restarts, network flow input records/sec
Cilium agent/operator	CPU/memory usage, FS write throughput/latency, container restarts
Retina (optional)	CPU/memory usage, FS write behavior
Control plane	API server latency, Pod startup latency
Node disk	Write throughput/latency via node-exporter

What's included

ClusterLoader2 workload

modules/python/clusterloader2/scale/ - CL2 configuration and Python harness
Fortio traffic generation (configurable deployments, replicas, QPS)
ContainerNetworkLog CRD deployment with filters for traffic namespaces
NetworkPolicy object scale (for API/controller load testing)

Pipeline & scenario

pipelines/perf-eval/CNI Benchmark/cnl-observability.yml - Pipeline with parameters for traffic scale and CNL options
scenarios/perf-eval/cnl-azurecni-overlay-cilium/ - Terraform inputs for AKS with:
- Kubernetes 1.33 (required for CNL)
- Azure CNI overlay + Cilium dataplane
- ACNS enabled (--enable-acns)
- 1000-node traffic pool + dedicated Prometheus pool

Monitoring

Node-exporter DaemonSet with ServiceMonitor
AMA-logs PodMonitor
Hubble metrics PodMonitor (scrapes cilium-agent port 9965)

Prerequisites

Per CNL documentation:

Azure CLI 2.75.0+ with aks-preview extension 19.0.07+
AdvancedNetworkingFlowLogsPreview feature flag registered
Kubernetes 1.33+
Cilium dataplane (for stored logs mode)

Notes

The NetworkPolicy template uses dummy selectors for API/etcd object scale testing only—it does not affect dataplane traffic
Node-exporter requires hostNetwork/hostPID for host-level metric collection

Copilot

Pull request overview

Adds a new observability perf-eval topology and a ClusterLoader2 scale workload aimed at measuring Container Network Logs (CNL/ACNS) overhead on AKS + Azure CNI Overlay + Cilium, including Prometheus scraping objects and result collection into JSONL.

Changes:

Introduces a new observability topology (validate/execute/collect) that runs a new CL2 “scale” engine workflow.
Adds a new CL2 “scale” harness (scale.py) plus CL2 configs/modules for Fortio traffic, CNL CRD creation, and Prometheus queries (AMA Logs / Cilium / node disk / control plane / Retina optional).
Adds a new Terraform scenario input set for a 1000-node AKS cluster with Azure CNI overlay + Cilium + ACNS enabled.

Reviewed changes

Copilot reviewed 34 out of 34 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
steps/topology/observability/validate-resources.yml	New topology validation step (currently only kubeconfig).
steps/topology/observability/execute-clusterloader2.yml	Wires observability topology to the CL2 scale engine execute template.
steps/topology/observability/collect-clusterloader2.yml	Wires observability topology to the CL2 scale engine collect template.
steps/engine/clusterloader2/scale/execute.yml	New CL2 scale execution template (includes optional Retina install).
steps/engine/clusterloader2/scale/collect.yml	New CL2 scale collection template producing JSONL results.
scenarios/perf-eval/cnl-azurecni-overlay-cilium/terraform-test-inputs/azure.json	Scenario test inputs for Azure terraform validation.
scenarios/perf-eval/cnl-azurecni-overlay-cilium/terraform-inputs/azure.tfvars	AKS 1.33 + Azure CNI overlay + Cilium + ACNS + 1000-node pool scenario definition.
modules/python/clusterloader2/scale/scale.py	New Python harness to configure/execute/collect CL2 scale runs and emit JSONL.
modules/python/clusterloader2/scale/config/modules/test-steps.yaml	CL2 workload steps (Fortio deploys, optional label churn, netpol scale, sleep window, cleanup).
modules/python/clusterloader2/scale/config/modules/scale-test.yaml	Orchestrates measurement modules + test steps (start/gather).
modules/python/clusterloader2/scale/config/modules/pfl/retinanetworkflowlog.yaml	Template for creating ContainerNetworkLog CRs (for flow logging).
modules/python/clusterloader2/scale/config/modules/node-exporter/servicemonitor.yaml	Prometheus ServiceMonitor for node-exporter.
modules/python/clusterloader2/scale/config/modules/node-exporter/serviceaccount.yaml	ServiceAccount for node-exporter components.
modules/python/clusterloader2/scale/config/modules/node-exporter/service.yaml	Service for node-exporter/kube-rbac-proxy endpoint.
modules/python/clusterloader2/scale/config/modules/node-exporter/networkpolicy.yaml	NetworkPolicy allowing Prometheus to scrape node-exporter.
modules/python/clusterloader2/scale/config/modules/node-exporter/daemonset.yaml	node-exporter + kube-rbac-proxy DaemonSet (hostNetwork/hostPID).
modules/python/clusterloader2/scale/config/modules/node-exporter/clusterrolebinding.yaml	RBAC binding for kube-rbac-proxy authn/authz checks.
modules/python/clusterloader2/scale/config/modules/node-exporter/clusterrole.yaml	ClusterRole for token/access reviews (kube-rbac-proxy).
modules/python/clusterloader2/scale/config/modules/node-exporter.yaml	CL2 module wrapper to create/delete node-exporter stack.
modules/python/clusterloader2/scale/config/modules/networkpolicy-template.yaml	Dummy NetworkPolicy template for API/etcd object scale testing.
modules/python/clusterloader2/scale/config/modules/measurements/retina.yaml	Prometheus queries for Retina resource + FS metrics.
modules/python/clusterloader2/scale/config/modules/measurements/node-disk.yaml	Prometheus queries for node disk write throughput/latency.
modules/python/clusterloader2/scale/config/modules/measurements/control-plane.yaml	Control plane queries (API responsiveness, pod startup latency, apiserver CPU/mem).
modules/python/clusterloader2/scale/config/modules/measurements/cilium.yaml	Prometheus queries for Cilium agent/operator CPU/mem/FS/restarts.
modules/python/clusterloader2/scale/config/modules/measurements/ama-logs.yaml	Prometheus queries for AMA Logs CPU/mem/FS/restarts + flow pipeline rates.
modules/python/clusterloader2/scale/config/modules/hubble/podmonitor.yaml	PodMonitor for scraping Hubble metrics via cilium-agent port 9965.
modules/python/clusterloader2/scale/config/modules/hubble.yaml	CL2 module wrapper to create/delete the Hubble PodMonitor.
modules/python/clusterloader2/scale/config/modules/fortio/service.yaml	Fortio service template used by the workload.
modules/python/clusterloader2/scale/config/modules/fortio/server-deployment.yaml	Fortio server deployment template (nodeSelector scale-test=true).
modules/python/clusterloader2/scale/config/modules/fortio/client-deployment.yaml	Fortio client deployment template (sustained load).
modules/python/clusterloader2/scale/config/modules/ama-logs/podmonitor.yaml	PodMonitor for scraping AMA Logs metrics in kube-system.
modules/python/clusterloader2/scale/config/modules/ama-logs.yaml	CL2 module wrapper to create/delete the AMA Logs PodMonitor.
modules/python/clusterloader2/scale/config/config.yaml	Top-level CL2 config wiring node-exporter/AMA/hubble + scale-test module.

steps/engine/clusterloader2/scale/collect.yml

modules/python/clusterloader2/scale/scale.py

steps/topology/observability/validate-resources.yml

steps/topology/observability/execute-clusterloader2.yml

steps/engine/clusterloader2/scale/execute.yml

Copilot

Pull request overview

Copilot reviewed 34 out of 34 changed files in this pull request and generated 9 comments.

modules/python/clusterloader2/scale/scale.py

modules/python/clusterloader2/scale/config/modules/measurements/retina.yaml

modules/python/clusterloader2/scale/config/modules/measurements/cilium.yaml

scenarios/perf-eval/cnl-azurecni-overlay-cilium/terraform-inputs/azure.tfvars

modules/python/clusterloader2/scale/config/modules/measurements/ama-logs.yaml

modules/python/clusterloader2/scale/config/modules/measurements/cilium.yaml

Copilot

Pull request overview

Copilot reviewed 34 out of 34 changed files in this pull request and generated 9 comments.

steps/engine/clusterloader2/scale/execute.yml

modules/python/clusterloader2/scale/scale.py

modules/python/clusterloader2/scale/config/modules/node-exporter/daemonset.yaml

modules/python/clusterloader2/scale/config/modules/node-exporter/serviceaccount.yaml

steps/engine/clusterloader2/scale/execute.yml

steps/topology/observability/collect-clusterloader2.yml

…s-scale-test

vittoriasalim · 2026-02-13T05:31:38Z

steps/topology/observability/collect-clusterloader2.yml

+    engine_input: ${{ parameters.engine_input }}
+    region: ${{ parameters.regions[0] }}
+- script: |
+    if [ -n "$RUN_ID" ]; then


use set-run-id template to avoid repetition https://github.com/Azure/telescope/blob/main/steps/set-run-id.yml

vittoriasalim · 2026-02-13T05:34:17Z

modules/python/clusterloader2/utils.py

    if file_name.startswith(PROM_QUERY_PREFIX):
-        group_name = file_name.split("_")[1]
-        measurement_name = file_name.split("_")[0][len(PROM_QUERY_PREFIX)+1:]
+        parts = file_name.split("_")


This change might break other pipeline not all measurement have same length as this test

why the original approach does not work in your current test

The original approach assumed a fixed format like PromQuery__.json, but our new test files use a different naming convention with additional segments.

I've add defensive parsing that handles both formats - we can check the number of segments before parsing and fall back to the original logic for files that match the existing pattern. This way we won't break other pipelines.

vittoriasalim · 2026-02-17T01:29:58Z

scenarios/perf-eval/cnl-azurecni-overlay-cilium/terraform-inputs/azure.tfvars

+      },
+      {
+        name                 = "traffic"
+        node_count           = 1000


Is this in telescope sub, do we have enough quota for this test

vittoriasalim · 2026-02-23T03:34:22Z

steps/provision-resources.yml

+    set -eo pipefail
+    echo "Checking AdvancedNetworkingFlowLogsPreview feature flag status..."
+
+    feature_state=$(az feature show --namespace "Microsoft.ContainerService" --name "AdvancedNetworkingFlowLogsPreview" --query "properties.state" -o tsv 2>/dev/null || echo "NotRegistered")


This is not required, we simply just need to register this to the subscription you want to test and set that in the pipeline.

Not all test need to register this feature, this change will affect other pipelines test. I suggest to just enable them in your sub .

carlotaarvela force-pushed the acns-scale-test branch 7 times, most recently from 6eccab9 to ac56435 Compare February 11, 2026 18:15

carlotaarvela marked this pull request as ready for review February 11, 2026 18:16

carlotaarvela requested review from alyssa1303, anson627 and sumanthreddy29 as code owners February 11, 2026 18:16

Copilot AI review requested due to automatic review settings February 11, 2026 18:16

carlotaarvela requested review from vittoriasalim, wonderyl and xinWeiWei24 as code owners February 11, 2026 18:16

Copilot started reviewing on behalf of carlotaarvela February 11, 2026 18:16 View session

Copilot AI reviewed Feb 11, 2026

View reviewed changes

carlotaarvela force-pushed the acns-scale-test branch 2 times, most recently from 4a17573 to c839c20 Compare February 12, 2026 11:14

carlotaarvela requested a review from Copilot February 12, 2026 11:15

Copilot started reviewing on behalf of carlotaarvela February 12, 2026 11:15 View session

Copilot AI reviewed Feb 12, 2026

View reviewed changes

carlotaarvela force-pushed the acns-scale-test branch from c839c20 to 93b0ab5 Compare February 12, 2026 11:46

carlotaarvela requested a review from Copilot February 12, 2026 11:47

Copilot started reviewing on behalf of carlotaarvela February 12, 2026 11:48 View session

Copilot AI reviewed Feb 12, 2026

View reviewed changes

Add ACNS observability test with Cilium dataplane

1a389c0

carlotaarvela force-pushed the acns-scale-test branch from 93b0ab5 to 1a389c0 Compare February 12, 2026 12:46

carlotaarvela and others added 2 commits February 12, 2026 12:46

Merge branch 'main' into acns-scale-test

30540ff

Fix test

d11d8a3

carlotaarvela added 2 commits February 12, 2026 13:11

Merge branch 'acns-scale-test' of github.com:Azure/telescope into acn…

5e96faa

…s-scale-test

Fix lint!

85c2f34

carlotaarvela force-pushed the acns-scale-test branch 2 times, most recently from 119a9d7 to 85c2f34 Compare February 12, 2026 15:20

vittoriasalim reviewed Feb 13, 2026

View reviewed changes

carlotaarvela added 2 commits February 13, 2026 15:22

Update clusterloader2 scale config and observability pipeline

78b09db

Fix trailing spaces lint errors

c3cb269

carlotaarvela requested a review from vittoriasalim February 13, 2026 15:32

vittoriasalim reviewed Feb 17, 2026

View reviewed changes

vittoriasalim reviewed Feb 23, 2026

View reviewed changes

Conversation

carlotaarvela commented Feb 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What's measured

What's included

ClusterLoader2 workload

Pipeline & scenario

Monitoring

Prerequisites

Notes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vittoriasalim Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

carlotaarvela Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

vittoriasalim Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

carlotaarvela Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

vittoriasalim Feb 17, 2026

Choose a reason for hiding this comment

Uh oh!

vittoriasalim Feb 23, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

carlotaarvela commented Feb 11, 2026 •

edited

Loading