Failed to get the ignition-clevis-pin-trustee file

[fangge1212]

I had this problem when trying to install the operator as static pod on k8s node.

Steps:

1. Build the operator images from latest code and push to quay.io:
   make REGISTRY=quay.io/rhn_support_fjin PUSH_FLAGS="--tls-verify=false" push

2. Create the manifests:
   make REGISTRY=quay.io/rhn_support_fjin manifests

3. Install k8s on a fedora vm

4. On the k8s node, copy these pod yaml files to /etc/kubernetes/manifests: trustee_deployment.yaml, cocl-operator.yaml, register-server.yaml. They will be started as static pods by kubelet service

5. Apply the CRD and CR
   kubectl apply -f trusted-execution-clusters.io_machines.yaml
kubectl apply -f trusted-execution-clusters.io_trustedexecutionclusters.yaml
kubectl apply -f trusted_execution_cluster_cr.yaml

6. Try to get the ignition file
   $ curl -O http://192.168.122.53:8000/ignition-clevis-pin-trustee
   $ cat ignition-clevis-pin-trustee
   {"code":500,"message":"Failed to get Trustee address: More than one TrustedExecutionCluster found in namespace default. trusted-cluster-operator does not support more than one TrustedExecutionCluster. Cancelling Ignition Clevis PIN request."}

7. Check the CR
   $ kubectl get trustedexecutionclusters.trusted-execution-clusters.io -A
   NAMESPACE NAME AGE
   trusted-execution-clusters trusted-execution-cluster 23h

[Jakob-Naucke]

There's a bug here, and it's register-server saying there was more than one TEC when there were zero. However, your namespaces are still mismatched: The TEC object is in namespace trusted-execution-clusters, but the register-server was in namespace default.

[fangge1212]

There's a bug here, and it's register-server saying there was more than one TEC when there were zero. However, your namespaces are still mismatched: The TEC object is in namespace trusted-execution-clusters, but the register-server was in namespace default.

My register server is in namespace trusted-execution-clusters:
$ kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
...skipped...
trusted-execution-clusters register-server-cp-0 1/1 Running 0 39h
trusted-execution-clusters trusted-cluster-operator-cp-0 1/1 Running 0 40h
trusted-execution-clusters trustee-deployment-cp-0 1/1 Running 0 39h

[Jakob-Naucke]

Oh. Do static pods default to the default namespace instead of their own when making API requests? Let me check.

[Jakob-Naucke]

@fangge1212 forgot to ask earlier: could you paste your static pod files here?

[fangge1212]

cocl-operator.yaml

apiVersion: v1
kind: Pod
metadata:
  name: trusted-cluster-operator
  namespace: trusted-execution-clusters
  labels:
    component: trusted-cluster-operator
    tier: control-plane
spec:
  hostNetwork: true
  containers:
  - name: trusted-cluster-operator
    command:
    - /usr/bin/operator
    image: quay.io/rhn_support_fjin/trusted-cluster-operator:latest
    imagePullPolicy: IfNotPresent
    env:
    - name: KUBECONFIG
      value: /etc/kubernetes/admin.conf
    resources: {}
    securityContext:
      seccompProfile:
        type: RuntimeDefault
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /etc/kubernetes/admin.conf
      name: kubeconfig
      readOnly: true
  volumes:
  - hostPath:
      path: /etc/kubernetes/admin.conf
      type: FileOrCreate
    name: kubeconfig

register-server.yaml

apiVersion: v1
kind: Pod
metadata:
  name: register-server
  namespace: trusted-execution-clusters
  labels:
    component: register-server
    tier: control-plane
spec:
  hostNetwork: true
  containers:
  - name: register-server
    args:
    - --port
    - "8000"
    image: quay.io/rhn_support_fjin/registration-server:latest
    imagePullPolicy: IfNotPresent
    env:
    - name: KUBECONFIG
      value: /etc/kubernetes/admin.conf
    ports:
    - containerPort: 8000
      name: http
      protocol: TCP
    resources: {}
    securityContext:
      seccompProfile:
        type: RuntimeDefault
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /etc/kubernetes/admin.conf
      name: kubeconfig
      readOnly: true
  volumes:
  - hostPath:
      path: /etc/kubernetes/admin.conf
      type: FileOrCreate
    name: kubeconfig

trustee-deployment.yaml

apiVersion: v1
kind: Pod
metadata:
  name: trustee-deployment
  namespace: trusted-execution-clusters
  labels:
    component: kbs
    tier: control-plane
spec:
  hostNetwork: true
  containers:
  - name: kbs
    command:
    - /usr/local/bin/kbs
    - --config-file
    - /opt/trustee/kbs-config.toml
    env:
    - name: RUST_LOG
      value: debug
    image: quay.io/trusted-execution-clusters/key-broker-service:tpm-verifier-built-in-as-20250711
    imagePullPolicy: IfNotPresent
    ports:
    - containerPort: 8080
      protocol: TCP
    resources: {}
    securityContext:
      seccompProfile:
        type: RuntimeDefault
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    # NOTE: Static pods cannot use ConfigMaps or Secrets.
    # You must create these directories on the host and mount them via hostPath,
    # or bake the configuration into the container image.
    # The original pod had 60+ ConfigMap and Secret mounts that need to be
    # replaced with hostPath volumes pointing to files on the node.
    - mountPath: /opt/trustee/policies/opa
      name: attestation-policy
    - mountPath: /opt/trustee
      name: trustee-data
    - mountPath: /opt/trustee/kbs-repository/default
      name: resource-dir
    - mountPath: /etc/kubernetes/admin.conf
      name: kubeconfig
      readOnly: true
  volumes:
  - name: attestation-policy
    hostPath:
      path: /var/lib/kubelet/static-pod-resources/trustee/attestation-policy
      type: DirectoryOrCreate
  - name: trustee-data
    hostPath:
      path: /var/lib/kubelet/static-pod-resources/trustee/trustee-data
      type: DirectoryOrCreate
  - name: resource-dir
    emptyDir:
      medium: Memory
  - name: kubeconfig
    hostPath:
      path: /etc/kubernetes/admin.conf
      type: FileOrCreate

trusted_execution_clustesr_cr.yaml

apiVersion: trusted-execution-clusters.io/v1alpha1
kind: TrustedExecutionCluster
metadata:
  name: trusted-execution-cluster
  namespace: trusted-execution-clusters
spec:
  pcrsComputeImage: quay.io/rhn_support_fjin/compute-pcrs:latest
  registerServerImage: quay.io/rhn_support_fjin/registration-server:latest
  trusteeImage: quay.io/trusted-execution-clusters/key-broker-service:tpm-verifier-built-in-as-20250711
  publicTrusteeAddr: 192.168.122.53:8080

[Jakob-Naucke]

Do static pods default to the default namespace instead of their own when making API requests?

Yes! And the namespace isn't in /var/run. You could query the API for pods, but without knowing your own name, let alone UID, you'll have ambiguities. However, we could pass that info via the downward API

e.g.

spec:
  containers:
  - …
    env:
    - name: BOOTSTRAP_NAMESPACE
      valueFrom:
        fieldRef:
          fieldPath: metadata.namespace

@fangge1212 does that sound reasonable?

[fangge1212]

Do static pods default to the default namespace instead of their own when making API requests?

Yes! And the namespace isn't in /var/run. You could query the API for pods, but without knowing your own name, let alone UID, you'll have ambiguities. However, we could pass that info via the downward API

e.g.

spec:
  containers:
  - …
    env:
    - name: BOOTSTRAP_NAMESPACE
      valueFrom:
        fieldRef:
          fieldPath: metadata.namespace

@fangge1212 does that sound reasonable?

Yes, I think it is reasonable