From 3fab34c39538d7790df3a3c28e23ff4f36bbc8da Mon Sep 17 00:00:00 2001 From: Harshit Gupta Date: Sat, 15 Nov 2025 11:47:03 +0100 Subject: [PATCH 1/2] identity plugin rfc Signed-off-by: Harshit Gupta --- docs/rfcs/0001-identity-plugin.md | 198 ++++++++++++++++++++++++++++++ 1 file changed, 198 insertions(+) create mode 100644 docs/rfcs/0001-identity-plugin.md diff --git a/docs/rfcs/0001-identity-plugin.md b/docs/rfcs/0001-identity-plugin.md new file mode 100644 index 00000000..c0016709 --- /dev/null +++ b/docs/rfcs/0001-identity-plugin.md @@ -0,0 +1,198 @@ +- Created: +- Current Version: +- Status: WIP +- Contributors: Harshit Gupta (hg2t4e@gmail.com, harshit.gupta@ibm.com), Michael Brown (brownwm@us.ibm.com) +- Approvers: + +# Summary + +The identity plugin, running as a daemonset, fetches identity artifacts for each annotated container and mounts the artifact volume into the container. + +# Motivation + +The current way how we mount identity artifacts into a container is to use a initContainer / sidecar. There is a sidecar for each pod. This sidecar fetches the artifacts from identity providers, injects the volume containing identity artifacts into the pod and mounts the volume into the container. Using a sidecar in each pod for fetching identity artifacts is too resource intensive. Using a initContainer sidecar comes with an additional overhead of increased container start latency. The motivation for using a NRI Identity Plugin is that only one process is enough to fetch the identity artifacts for all of the annotated containers in all the pods on a single host. Therefore the identity PLUGIN as a daemonset can fetch the identity artifacts for all the containers in all the pods on all thee hosts in the cluster that have the appropriate annotations. Running as a single process per host instead of single process per pod we save on host resources. + +# Proposed Implementation + +## Setup Steps + +1. Create a static spiffe-id for the plugin. + +``` +spire-server entry create \ + -spiffeID spiffe://example.org/nri/identity-injector \ + -selector k8s:ns:backend \ + -selector k8s:sa:db-writer +``` + +PENDING - what selector's will the plugin have? + + +2. https://spiffe.io/docs/latest/deploying/spire_agent/#delegated-identity-api Configure the Plugin's Spiffe ID in the Agent configuration. + +``` +agent { + trust_domain = "example.org" + ... + admin_socket_path = "/tmp/spire-agent/private/admin.sock" + authorized_delegates = [ + "spiffe://example.org/nri/identity-injector" + ] +} + +``` + + +## How the plugin will work + +1. Plugin registers itself to the agent and get its spiffe id. + +2. Containerd calls the NRI plugin on CreateContainer event. The plugin receives the pod and container metadata. Using this metadata, the plugin executes step 3 below. The plugin has to ensure that the pid is a stable identifier, that the pid is not a recycled identifier. This is a platform dependent step. (e.g. by using pidfds on Linux). https://spiffe.io/docs/latest/deploying/spire_agent/#delegated-identity-api + +3. The plugin reads the podspec and parses the annotations. Using spire-api-sdk, the plugin fetches identity artifacts and mounts the volume containing the identity artifacts into the container. The spire-api-sdk streams artifact updates, the plugin will on receiving updates will overwrite the artifact on the host volume. Since the host volume is mounted into the container, the artifacts in the container are automatically updated. The plugin does not need a separate mount annotation in the podspec yaml. The daemonset specifies the volume that the plugin will use to fetch and download the identity artifacts. The plugin will then mount the host volume to the container volume using the cert_dir from the podspec. + + + +## DaemonSet yaml + +``` +apiVersion: apps/v1 +kind: DaemonSet +metadata: + name: nri-plugin-identity-injector +spec: + template: + spec: + priorityClassName: system-node-critical + containers: + - name: plugin + image: plugin:latest + args: + - "-idx" + - "10" + resources: + requests: + cpu: "2m" + memory: "5Mi" + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL + volumeMounts: + - name: nri-socket + mountPath: /var/run/nri/nri.sock + - name: identity-artifacts # Artifact volume + mountPath: /var/identity/ + volumes: + - name: nri-socket + hostPath: + path: /var/run/nri/nri.sock + type: Socket + - name: identity-artifacts # Artifact volume + hostPath: + path: /var/identity/ + type: Directory + +``` + + + +## Podspec example + +``` +apiVersion: v1 +kind: Pod +metadata: + name: bbid0 + labels: + app: bbid0 + annotations: + identity.noderesource.dev/container.c0: |+ + - spire_agent_address: /tmp/agent.sock + cert_dir: /var/certs + svid_file_name: svid.pem + svid_key_file_name: svid_key.pem + svid_bundle_file_name: svid_bundle.pem + identity.noderesource.dev/container.c1: |+ + - spire_agent_address: /tmp/agent.sock + cert_dir: /var/certs + svid_key_file_name: svid_key.pem + svid_bundle_file_name: svid_bundle.pem +spec: + containers: + - name: c0 + image: busybox + imagePullPolicy: IfNotPresent + command: + - sh + - -c + - | + if [ -f /var/certs/svid.pem ]; then + echo "svid exists!" + else + echo "svid does NOT exist." + fi + sleep inf + resources: + requests: + cpu: 500m + memory: '100M' + limits: + cpu: 500m + memory: '100M' + - name: c1 + image: busybox + imagePullPolicy: IfNotPresent + command: + - sh + - | + if [ -f /var/certs/svid.pem ]; then + echo "svid exists!" + else + echo "svid does NOT exist." + fi + sleep inf + resources: + requests: + cpu: 1 + memory: '100M' + limits: + cpu: 1 + memory: '100M' + - name: c2 + image: busybox + imagePullPolicy: IfNotPresent + command: + - sh + - -c + - | + if [ -f /var/certs/svid.pem ]; then + echo "svid exists!" + else + echo "svid does NOT exist." + fi + sleep inf + resources: + requests: + cpu: 1 + memory: '100M' + limits: + cpu: 1 + memory: '100M' + terminationGracePeriodSeconds: 1 + +``` + +## Handling Openshift / cri-o + +Openshift / cri-o specific challenges that the plugin resolves. + +## Handling VMs + + +# Alternatives + +An alternative to identity plugin is to use sidecar containers or initContainers in every pod for fetching the artifacts and mounting the artifact volume. + +# Open Questions \ No newline at end of file From 11e2a8feaf757e38c00dbb13094c79e983fd7a37 Mon Sep 17 00:00:00 2001 From: Harshit Gupta Date: Thu, 27 Nov 2025 10:59:35 +0100 Subject: [PATCH 2/2] Add alternatives to NRI Plugin Signed-off-by: Harshit Gupta --- docs/rfcs/0001-identity-plugin.md | 51 ++++++++++++++++++++++++++++--- 1 file changed, 47 insertions(+), 4 deletions(-) diff --git a/docs/rfcs/0001-identity-plugin.md b/docs/rfcs/0001-identity-plugin.md index c0016709..ebf6d8a8 100644 --- a/docs/rfcs/0001-identity-plugin.md +++ b/docs/rfcs/0001-identity-plugin.md @@ -6,16 +6,18 @@ # Summary -The identity plugin, running as a daemonset, fetches identity artifacts for each annotated container and mounts the artifact volume into the container. +The identity plugin, running as a daemonset, fetches identity artifacts (X.509 SVID certificate bundle) for each annotated container and mounts the artifact volume into the container. # Motivation -The current way how we mount identity artifacts into a container is to use a initContainer / sidecar. There is a sidecar for each pod. This sidecar fetches the artifacts from identity providers, injects the volume containing identity artifacts into the pod and mounts the volume into the container. Using a sidecar in each pod for fetching identity artifacts is too resource intensive. Using a initContainer sidecar comes with an additional overhead of increased container start latency. The motivation for using a NRI Identity Plugin is that only one process is enough to fetch the identity artifacts for all of the annotated containers in all the pods on a single host. Therefore the identity PLUGIN as a daemonset can fetch the identity artifacts for all the containers in all the pods on all thee hosts in the cluster that have the appropriate annotations. Running as a single process per host instead of single process per pod we save on host resources. +The current way how we mount identity artifacts (X.509 SVID certificate bundle) into a container is to use a initContainer / sidecar. There is a sidecar for each pod. This sidecar fetches the artifacts from identity providers, injects the volume containing identity artifacts into the pod and mounts the volume into the container. Using a sidecar in each pod for fetching identity artifacts is too resource intensive. Using a initContainer sidecar comes with an additional overhead of increased container start latency. The motivation for using a NRI Identity Plugin is that only one process is enough to fetch the identity artifacts for all of the annotated containers in all the pods on a single host. Therefore the identity Plugin as a daemonset can fetch the identity artifacts for all the containers in all the pods on all thee hosts in the cluster that have the appropriate annotations. Running as a single process per host instead of single process per pod we save on host resources. # Proposed Implementation ## Setup Steps +WIP + 1. Create a static spiffe-id for the plugin. ``` @@ -45,9 +47,11 @@ agent { ## How the plugin will work +WIP + 1. Plugin registers itself to the agent and get its spiffe id. -2. Containerd calls the NRI plugin on CreateContainer event. The plugin receives the pod and container metadata. Using this metadata, the plugin executes step 3 below. The plugin has to ensure that the pid is a stable identifier, that the pid is not a recycled identifier. This is a platform dependent step. (e.g. by using pidfds on Linux). https://spiffe.io/docs/latest/deploying/spire_agent/#delegated-identity-api +2. Containerd calls the NRI plugin on CreateContainer event - PENDING fix the correct event, add details on when and which volume is mounted. The plugin receives the pod and container metadata. Using this metadata, the plugin executes step 3 below. The plugin has to ensure that the pid is a stable identifier, that the pid is not a recycled identifier. This is a platform dependent step. (e.g. by using pidfds on Linux). https://spiffe.io/docs/latest/deploying/spire_agent/#delegated-identity-api 3. The plugin reads the podspec and parses the annotations. Using spire-api-sdk, the plugin fetches identity artifacts and mounts the volume containing the identity artifacts into the container. The spire-api-sdk streams artifact updates, the plugin will on receiving updates will overwrite the artifact on the host volume. Since the host volume is mounted into the container, the artifacts in the container are automatically updated. The plugin does not need a separate mount annotation in the podspec yaml. The daemonset specifies the volume that the plugin will use to fetch and download the identity artifacts. The plugin will then mount the host volume to the container volume using the cert_dir from the podspec. @@ -55,6 +59,8 @@ agent { ## DaemonSet yaml +WIP + ``` apiVersion: apps/v1 kind: DaemonSet @@ -100,6 +106,11 @@ spec: ## Podspec example +WIP PENDING - are annotations immutable? can we support mutable annotations? +WIP PENDING - supporting pod certificate KEP 4317 + +WIP + ``` apiVersion: v1 kind: Pod @@ -188,11 +199,43 @@ spec: Openshift / cri-o specific challenges that the plugin resolves. +WIP + ## Handling VMs +WIP + # Alternatives +## Sidecars / InitContainers + An alternative to identity plugin is to use sidecar containers or initContainers in every pod for fetching the artifacts and mounting the artifact volume. -# Open Questions \ No newline at end of file +Advantage of using NRI Plugin over sidecars: SideCars / initContainers require additional resources. NRI Plugin does not need sidecar containers or initContainers therefore the NRI Plugin is resource efficient. + + +## CSI Driver + +Another alternative is to use a CSI Driver. The CSI Driver uses the Pod's PID to fetch identity artifacts before the container is created/started. + +Advantage of using NRI Plugin over CSI Driver: + - CSI drivers require extra sidecars for their functioning. Although these sidecars per node (not per pod), they still consume resources. NRI Plugin does not require any sidecars to support its function and is therefore resource efficient. + - Since the container is actually using the identity artifacts, the actual PID that the identity artifacts should be fetched for is the container PID. CSI Driver does not have the container PID at the time it fetches the identity artifacts therefore the CSI Driver uses the Pod PID. CSI Driver could use the container PID as soon as it is available after container start, but by that time a container might have executed commands without identity artifacts, breaking security. + +## Advantages of using NRI Identity Plugin over K8s KEP 4317 Pod Certificates + +WIP + +- Adds supports for stacks not using the Kubelet. +- Must also be using the Pod's PID to fetch identity artifacts. + + +## Advantages of using NRI Identity Plugin over Container Lifecycle Hooks + +WIP + + +# Open Questions + +How can the container runtime not fetch the certs if the kubelet is also fetching them? or the other way around \ No newline at end of file