Introduce MAC collision metric and alert

[RamLavi]

What this PR does / why we need it:
This PR introduces:

• a new gauge metric for kmp_mac_collisions, deployed by kubemacpool.
• a new alert for running objects MAC collision.
  It also references the new alert on the deprecated alert.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Special notes for your reviewer:
Assisted by Claude

Checklist

This checklist is not enforcing, but it's a reminder of items that could be relevant to every PR.
Approvers are expected to review this list.

• Design: A design document was considered and is present (link) or not required
• PR: The PR description is expressive enough and will help future contributors
• Code: Write code that humans can understand and Keep it simple
• Refactor: You have left the code cleaner than you found it (Boy Scout Rule)
• Upgrade: Impact of this change on upgrade flows was considered and addressed if required
• Testing: New code requires new unit tests. New features and bug fixes require at least on e2e test
• Documentation: A user-guide update was considered and is present (link) or not required. You want a user-guide update if it's a user facing feature / API change.
• Community: Announcement to kubevirt-dev was considered

Release note:

Introduce kmp_mac_collisions metric 
Introduce KubemacpoolMACCollisionDetected alert Runbook

[RamLavi]

/hold

until the alert is merged and bumped to CNAO

[RamLavi]

example of how the diagnoses should look like:

[root@zeus22 kubemacpool]# kubectl exec -n $KMP_NAMESPACE deployment/kubemacpool-mac-controller-manager \
+      -c manager -- curl -s http://localhost:8080/metrics | grep kmp_mac_collisions
# HELP kmp_mac_collisions Count of running objects sharing the same MAC address (collision when > 1)
# TYPE kmp_mac_collisions gauge
kmp_mac_collisions{mac="02:00:00:aa:bb:cc"} 2
[root@zeus22 kubemacpool]#
[root@zeus22 kubemacpool]#
[root@zeus22 kubemacpool]# kubectl get vmi -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"\t"}{.metadata.name}{"\t"}{.status.interfaces[*].mac}{"\n"}{end}' | grep -i "02:00:00:aa:bb:cc"
test-collision-ns1	test-vm-collision-1	02:00:00:aa:bb:cc
test-collision-ns2	test-vm-collision-2	02:00:00:aa:bb:cc

[RamLavi]

/hold cancel

@sradco all feature content has been merged. We can proceed with this PR.
Could you take a look?

[RamLavi]

Hey @avlitman is the failed sanity unrelated? The runbooks there are not related to the one I am adding.

[avlitman]

Hey @avlitman is the failed sanity unrelated? The runbooks there are not related to the one I am adding.

fixed in #353 you can rebase

[RamLavi]

Change: Rebase

[avlitman]

this should be added automatically once it's added to the repo, so add kubemacpool here: https://github.com/kubevirt/monitoring/blob/main/tools/metricsdocs/types.go#L26 and run the workflow

[RamLavi]

DONE

had one comment

[avlitman]

Hi @sradco this runbook and metric is not under kubevirt org it's here: https://github.com/k8snetworkplumbingwg/kubemacpool/blob/main/pkg/monitoring/rules/alerts/alerts.go#L39

I wonder if it's the right place to add this runbook and metric documentation. WDTY?

[RamLavi]

Hi @sradco this runbook and metric is not under kubevirt org it's here: https://github.com/k8snetworkplumbingwg/kubemacpool/blob/main/pkg/monitoring/rules/alerts/alerts.go#L39

I wonder if it's the right place to add this runbook and metric documentation. WDTY?

IMO since this repo is deployed by HCO (by extention by CNAO) - we should treat it as one deployed by the product...
@sradco what do you say?

[RamLavi]

Change: Rebase

[sradco]

@RamLavi please add to the PR description the link to the PR of the new alert and metric. Thank you