feat(localdns): add systemd-based Prometheus-format metrics exporter

[saewoni]

Adds CPU and memory metrics for localdns.service using systemd accounting and socket activation for efficient, zero-overhead monitoring.

Also export IP addresses configured for forward plugins for kubednsoverrides and vnetdnsoverrides

Implementation:

• localdns_exporter.sh: Scrapes systemd CPUUsageNSec and MemoryCurrent
• localdns-exporter.socket: Socket activation on port 9353
• localdns-exporter@.service: Instantiated service per connection
• Integrated into all VHD builders (Ubuntu, Mariner, Flatcar, all arches)

Metrics exposed:

• localdns_cpu_usage_seconds_total (counter)
• localdns_memory_usage_mb (gauge)

Test coverage:

• e2e/test-localdns-exporter.sh validates exporter functionality

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #

[Copilot]

Pull request overview

This PR adds a lightweight Prometheus-compatible metrics exporter for the localdns.service using systemd socket activation. The exporter exposes CPU and memory metrics on port 9353 with zero overhead when not being scraped, making it suitable for production monitoring.

Changes:

• Added localdns_exporter.sh bash script that queries systemd accounting metrics and formats them as Prometheus metrics
• Added systemd socket (localdns-exporter.socket) and service (localdns-exporter@.service) units for on-demand activation
• Integrated the exporter into all Linux VHD builds (Ubuntu, Mariner, Flatcar, ARM64/x64)
• Added VHD content validation tests and a standalone test script

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
parts/linux/cloud-init/artifacts/localdns_exporter.sh	Bash script that scrapes systemd metrics (CPUUsageNSec, MemoryCurrent) and outputs Prometheus-formatted HTTP response
parts/linux/cloud-init/artifacts/localdns-exporter@.service	Systemd template service for per-connection worker instances with security hardening
parts/linux/cloud-init/artifacts/localdns-exporter.socket	Systemd socket unit listening on port 9353 with Accept=yes for on-demand activation
vhdbuilder/packer/vhd-image-builder-*.json	Added file provisioners to copy exporter artifacts to all Linux VHD variants
vhdbuilder/packer/packer_source.sh	Added file copying logic and systemctl enable command for the socket
vhdbuilder/packer/imagecustomizer/azlosguard/azlosguard.yml	Added exporter files to OSGuard VHD build but missing socket enablement
vhdbuilder/packer/test/linux-vhd-content-test.sh	Added validation for exporter files and permissions
e2e/test-localdns-exporter.sh	Standalone test script for manual validation

[yewmsft]

how to we test these?

[yewmsft]

LGTM in general. Let's make sure we have proper test coverages

[Copilot]

Pull request overview

Copilot reviewed 24 out of 24 changed files in this pull request and generated 3 comments.

[Copilot]

This change updates parts/linux/cloud-init/artifacts/cse_config.sh, which is snapshot-tested via pkg/agent/testdata/** (CustomData/CSECommand). The PR currently doesn’t include regenerated testdata artifacts; make generate should be run and the updated snapshot files committed, otherwise unit tests comparing generated CustomData/CSE output are likely to fail in CI.

[Copilot]

The forward-IP extraction will incorrectly treat the '{' token as an IP when the corefile uses the common CoreDNS syntax forward . <ip> { ... } (the localdns corefile templates in this repo include that form). This will emit a bogus metric line with ip="{" (and potentially other non-IP tokens if the format changes). Update the parser to only capture actual IPs (e.g., stop at {/; or filter tokens by an IPv4/IPv6 regex) and add/adjust tests to cover the brace form.

Suggested change

	vnetdns_ips=($(awk '/bind 169.254.10.10/,/^}/' "${UPDATED_LOCALDNS_CORE_FILE}" | awk '/forward \. / {for(i=3; i<=NF; i++) print $i}'))
	kubedns_ips=($(awk '/bind 169.254.10.11/,/^}/' "${UPDATED_LOCALDNS_CORE_FILE}" | awk '/forward \. / {for(i=3; i<=NF; i++) print $i}'))
	vnetdns_ips=($(awk '/bind 169.254.10.10/,/^}/' "${UPDATED_LOCALDNS_CORE_FILE}" | awk '/forward \. / {
	for (i = 3; i <= NF; i++) {
	if ($i == "{" || $i == ";") break;
	# IPv4 address
	if ($i ~ /^([0-9]{1,3}\.){3}[0-9]{1,3}$/) {
	print $i;
	# Very loose IPv6 match: contains a colon and no braces/semicolons
	} else if ($i ~ /:/ && $i !~ /[{};]/) {
	print $i;
	}
	}
	}'))
	kubedns_ips=($(awk '/bind 169.254.10.11/,/^}/' "${UPDATED_LOCALDNS_CORE_FILE}" | awk '/forward \. / {
	for (i = 3; i <= NF; i++) {
	if ($i == "{" || $i == ";") break;
	# IPv4 address
	if ($i ~ /^([0-9]{1,3}\.){3}[0-9]{1,3}$/) {
	print $i;
	# Very loose IPv6 match: contains a colon and no braces/semicolons
	} else if ($i ~ /:/ && $i !~ /[{};]/) {
	print $i;
	}
	}
	}'))

[Copilot]

The test corefile fixtures here use forward . 168.63.129.16 without the { ... } options block, but the real localdns corefile template in this repo uses forward . <ip> { ... }. As written, these tests won’t catch the exporter bug where the { token gets exported as an IP. Consider updating at least one fixture to include forward . 168.63.129.16 { (plus a closing }) and assert that { is not emitted as an ip label.

[Copilot]

Pull request overview

Copilot reviewed 24 out of 24 changed files in this pull request and generated 1 comment.

[Copilot]

ListenStream=127.0.0.1:9353 already restricts the exporter to loopback. IPAddressAllow=localhost / IPAddressDeny=any is redundant here and can be misleading (it does not affect inbound socket binding). Consider removing these directives or adjusting the comment to reflect what they actually enforce.

Suggested change

	# Defense-in-depth: Restrict access to loopback only
	IPAddressAllow=localhost
	IPAddressDeny=any

[Copilot]

Pull request overview

Copilot reviewed 41 out of 89 changed files in this pull request and generated 4 comments.

[Copilot]

The forward_ips.prom generation is treated as fatal: any mktemp/chmod/mv failure returns non-zero, which causes replace_azurednsip_in_corefile to fail and the localdns service to exit (ERR_LOCALDNS_FAIL). Since these metrics are optional observability, this should be best-effort (log a warning and continue) so local DNS resolution isn’t taken down by an exporter artifact write failure.

[Copilot]

This script curls http://localhost:9353/metrics, but the socket is bound to 127.0.0.1:9353. On systems where localhost resolves to ::1 first, curl may attempt IPv6 and fail with HTTP code 000 even though the exporter is healthy. Use http://127.0.0.1:9353/metrics (or force IPv4 with curl -4) to match the ListenStream binding.

[Copilot]

This runtime hardening check depends on lsof, but lsof isn’t installed on all supported images (it’s installed on Mariner via cse_install_mariner.sh, but not obviously on Ubuntu/Flatcar). If lsof is missing, the script will fail and break the e2e scenario. Consider using /proc/$PID/fd + readlink/ss -xp for socket inspection, or gate this check on command -v lsof (skip with a warning when unavailable).

Suggested change

	NETWORK_SOCKETS=$(lsof -p "$INSTANCE_PID" 2>/dev/null | grep -c "IPv4\|IPv6" || echo "0")
	if [ "$NETWORK_SOCKETS" != "0" ]; then
	echo " ❌ ERROR: Instance has network sockets (RestrictAddressFamilies not enforced)"
	lsof -p "$INSTANCE_PID" | grep "IPv" || true
	exit 1
	fi
	echo " ✓ No network sockets (AF_UNIX only, restriction enforced)"
	if command -v lsof >/dev/null 2>&1; then
	NETWORK_SOCKETS=$(lsof -p "$INSTANCE_PID" 2>/dev/null | grep -c "IPv4\|IPv6" || echo "0")
	if [ "$NETWORK_SOCKETS" != "0" ]; then
	echo " ❌ ERROR: Instance has network sockets (RestrictAddressFamilies not enforced)"
	lsof -p "$INSTANCE_PID" | grep "IPv" || true
	exit 1
	fi
	echo " ✓ No network sockets (AF_UNIX only, restriction enforced)"
	else
	echo " ⚠️ WARNING: 'lsof' not found; skipping network socket verification (RestrictAddressFamilies runtime enforcement)"
	fi

[Copilot]

The security-directive assertions are brittle across OS/systemd versions: systemctl show --property=... may omit unsupported properties (and the unit may ignore unknown directives), but the script hard-fails expecting all 16 keys (e.g., PrivateMounts=yes). To avoid false negatives across Ubuntu/Azure Linux/Flatcar variants, detect whether each property is supported before asserting, or validate by parsing the unit file instead of requiring systemctl show to return every property.

Suggested change

	# Fetch all security-related properties in batches (systemctl has limits)
	SECURITY_PROPS_1=$(systemctl show localdns-exporter@.service \
	--property=DynamicUser,PrivateTmp,ProtectSystem,ProtectHome,ReadOnlyPaths,NoNewPrivileges \
	2>/dev/null || true)
	SECURITY_PROPS_2=$(systemctl show localdns-exporter@.service \
	--property=ProtectKernelTunables,ProtectKernelModules,ProtectControlGroups,RestrictAddressFamilies \
	2>/dev/null || true)
	SECURITY_PROPS_3=$(systemctl show localdns-exporter@.service \
	--property=RestrictNamespaces,LockPersonality,RestrictRealtime,RestrictSUIDSGID,RemoveIPC,PrivateMounts \
	2>/dev/null || true)
	SECURITY_PROPS="$SECURITY_PROPS_1
	$SECURITY_PROPS_2
	$SECURITY_PROPS_3"
	echo " Retrieved security properties:"
	echo "$SECURITY_PROPS" | sed 's/^/ /'
	echo ""
	# Check all 16 security directives
	# Fetch all security-related properties individually so we can detect unsupported ones
	SECURITY_PROPERTIES=(
	DynamicUser
	PrivateTmp
	ProtectSystem
	ProtectHome
	ReadOnlyPaths
	NoNewPrivileges
	ProtectKernelTunables
	ProtectKernelModules
	ProtectControlGroups
	RestrictAddressFamilies
	RestrictNamespaces
	LockPersonality
	RestrictRealtime
	RestrictSUIDSGID
	RemoveIPC
	PrivateMounts
	)
	SECURITY_PROPS=""
	SUPPORTED_SECURITY_PROPS=()
	UNSUPPORTED_SECURITY_PROPS=()
	for prop in "${SECURITY_PROPERTIES[@]}"; do
	# systemctl show prints "<Prop>=<value>" when the property is known; for
	# unknown/unsupported properties it typically prints nothing.
	value="$(systemctl show localdns-exporter@.service --property="$prop" 2>/dev/null || true)"
	# Normalize whitespace and ignore empty output (treated as unsupported).
	if [[ -n "${value//[[:space:]]/}" ]]; then
	SECURITY_PROPS+="${value}"$'\n'
	SUPPORTED_SECURITY_PROPS+=("$prop")
	else
	UNSUPPORTED_SECURITY_PROPS+=("$prop")
	fi
	done
	echo " Retrieved security properties (supported on this systemd version):"
	echo "$SECURITY_PROPS" | sed 's/^/ /'
	if ((${#UNSUPPORTED_SECURITY_PROPS[@]} > 0)); then
	echo ""
	echo " Note: the following security directives are not supported by this systemd version and will be skipped:"
	for prop in "${UNSUPPORTED_SECURITY_PROPS[@]}"; do
	echo " - $prop"
	done
	fi
	echo ""
	# Check all 16 security directives (only those supported will have entries in SECURITY_PROPS)