Minio operator bootstrapped with bad certificate

[L1ghtman2k]

When starting a new minio operator chart with following values:

console:
  enabled: false
operator:
  image:
    repository: quay.io/minio/operator
    tag: v6.0.4
  replicaCount: 2
  resources:
    limits:
      cpu: 100m
      memory: 128Mi
    requests:
      cpu: 0
      memory: 0
  tolerations:
  - effect: NoSchedule
    key: node.kubernetes.io/node
    operator: Equal
    value: cloudops

I ran into an issue where both of the minio-operator replicas report:

panic: tls: private key does not match public key

goroutine 168 [running]:
github.com/minio/operator/pkg/controller.(*Controller).waitForCertSecretReady(0xc000a242c0, {0x210e1cc, 0x3}, {0x21104a3, 0x7})
        github.com/minio/operator/pkg/controller/tls.go:67 +0x2ec
github.com/minio/operator/pkg/controller.(*Controller).waitSTSTLSCert(...)
        github.com/minio/operator/pkg/controller/sts.go:421
github.com/minio/operator/pkg/controller.(*Controller).startSTSAPIServer(0xc000a242c0, {0x244fc30, 0xc000a7a050}, 0xc0001329c0)
        github.com/minio/operator/pkg/controller/main-controller.go:420 +0xab
created by github.com/minio/operator/pkg/controller.(*Controller).Start in goroutine 1
        github.com/minio/operator/pkg/controller/main-controller.go:563 +0x4db

The minio secret looks to have a malformed private key:

onprem_shell@onprem-node23:~$ kubectl get secret -n minio-operator sts-tls -o yaml
apiVersion: v1
data:
  private.key: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JR0hBZ0VBTUJNR0J5cUdTTTQ5QWdFR0NDcUdTTTQ5QXdFSEJHMHdhd0lCQVFRZzRweHlDbS94WmJmRENPR1AKMVR1N2dTb21CcFBxbHNNVHUwN2JrSGIzOFJHaFJBTkNBQVRpKzBSZ3FTV3ZQVkJTZ0FRZDZINjYxZUQrMExZcgpEMHIyWHdsSU9HdnFLclc0bVkydWRncjJ3RVh4M1dNOTRySlQxVit4TWVHOWhxSFhKVjVJdDJqcAotLS0tLUVORCBQUklWQVRFIEtFWS0tLS0tCg==
  public.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN4VENDQWEyZ0F3SUJBZ0lRY0NpOFV1aHBrSDhKdFgrb3JSOGhxekFOQmdrcWhraUc5dzBCQVFzRkFEQWMKTVJvd0dBWURWUVFEREJGVGFXeDJaWEpqY21WbGEwMXJPSE5EUVRBZUZ3MHlOVEEwTWpneE1URXdNRFJhRncweQpOakEwTWpNd09EQXdNamRhTUVReEZUQVRCZ05WQkFvVERITjVjM1JsYlRwdWIyUmxjekVyTUNrR0ExVUVBeE1pCmMzbHpkR1Z0T201dlpHVTZjM1J6TG0xcGJtbHZMVzl3WlhKaGRHOXlMbk4yWXpCWk1CTUdCeXFHU000OUFnRUcKQ0NxR1NNNDlBd0VIQTBJQUJHR3Z4ZW1iZ2hUN2k1K3QyWjVKcVpCVmcwRWZoaGdka0xseng1bTI2R2JnTmlWTwpyR29JemJEVHo2N2tBSU9Ua2hGOHpCS2RvNUV4WStaNEFOcGNLWStqZ2FVd2dhSXdEZ1lEVlIwUEFRSC9CQVFECkFnV2dNQk1HQTFVZEpRUU1NQW9HQ0NzR0FRVUZCd01CTUF3R0ExVWRFd0VCL3dRQ01BQXdId1lEVlIwakJCZ3cKRm9BVVhSNHUvU1B6cS9VS3RVeWNUb0RBVXU4aGhQWXdUQVlEVlIwUkJFVXdRNElEYzNSemdoWnpkSE11YldsdQphVzh0YjNCbGNtRjBiM0l1YzNaamdpUnpkSE11YldsdWFXOHRiM0JsY21GMGIzSXVjM1pqTG1Oc2RYTjBaWEl1CmJHOWpZV3d3RFFZSktvWklodmNOQVFFTEJRQURnZ0VCQUZnOUZ1ZVBKaGUzcVNTL2tjRUxDM1lDNWNXVHF3TXUKc3N0YlZnb0ZzYWw0c0lmVW5wZlo2SmZCcEtEdkNNZFdpczd0eU1YVlp6QUpuTEd3NkNzYWZSUEx2UW4ybjY2bwp5VnVhc2JEeTAzK2dzQW54Y2dONWdsbFErMEhlUW85Vkh4TjZnOVN3bUY4cHIvRVJRcEhNN1E5U3B0bmZJUmdSCnRBcC9lOHB5cGFBRFVZMkRTOHcxcmRNcTMwOWcySXFWYmltLzRnSEZONWRwWkM1d0ZJclhrMHh3bXBoMWJoaEcKNi9pT1NvRTBRaVFjR0prZmxSaFNzSGZ4bHZ1S1ZNWE5NLzBuVlJyeVRhNjRKV0lBTExVKzVQZEEwOTNwL0dnbgpHaUx1Z3NNM1hueFJUbDJTVm4zUjRUNGoySWRlQnlha3RlMy9JTkI3djVMbXN3UElFelFlRzZrPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
kind: Secret
metadata:
  creationTimestamp: "2025-04-28T11:15:08Z"
  name: sts-tls
  namespace: minio-operator
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: Deployment
    name: minio-operator
    uid: a696df15-f7ac-4539-a749-78056d0ba64d
  resourceVersion: "40373"
  uid: 6ae744cf-f472-4d7a-97f7-624608da9123
type: Opaque

The issues seemed to not have occurred right away. We deploy minio operator and a tenant right after. The tenant got deployed, but midway thru the deployment minio-operator started reporting this issue. This bug also doesn't manifest itself consistently.

Your Environment

• Version used (minio-operator): 6.0.4
• Environment name and version (e.g. kubernetes v1.17.2): k8s 1.31
• Server type and version: Vmware
• Operating System and version (uname -a): Ubuntu

[ramondeklein]

This typically happens when there is still an old CertificateSigningRequest that matches the name, but uses a different public/private key. You may want to delete the CSR and try again...

[L1ghtman2k]

We are deploying our clusters on a fresh k8s, and it seems like this issue happens after the operator have been running for some time

[ramondeklein]

Can you post fulll minio-operator logging and run kubectl get csr -n <ns> and post the output? A mismatch between private and public key only occurs when the CSR is created using a different key-pair. This can only happen if no certificate is issued for the private key and the public/private keys get out of sync.

[L1ghtman2k]

ran into this on the latest operator:

root@onprem-node245:/home/onprem_shell# kubectl logs minio-operator-c8b489f59-lcnkk -n minio-operator
I0530 04:09:50.213421       1 controller.go:81] Starting MinIO Operator
I0530 04:09:50.214924       1 main-controller.go:293] Setting up event handlers
I0530 04:09:50.514738       1 main-controller.go:514] Using Kubernetes CSR Version: v1
I0530 04:09:50.514778       1 main-controller.go:534] Waiting for STS API to start
I0530 04:09:50.514801       1 leaderelection.go:257] attempting to acquire leader lease minio-operator/minio-operator-lock...
I0530 04:09:50.611229       1 main-controller.go:399] Starting STS API server
I0530 04:09:50.714978       1 main-controller.go:595] minio-operator-c8b489f59-v8ts6: is the leader, removing any leader labels that I 'minio-operator-c8b489f59-lcnkk' might have
panic: tls: private key does not match public key

goroutine 103 [running]:
github.com/minio/operator/pkg/controller.(*Controller).waitForCertSecretReady(0xc00047ec80, {0x25c6127, 0x3}, {0x25c9b3f, 0x7})
        github.com/minio/operator/pkg/controller/tls.go:67 +0x2ea
github.com/minio/operator/pkg/controller.(*Controller).waitSTSTLSCert(...)
        github.com/minio/operator/pkg/controller/sts.go:427
github.com/minio/operator/pkg/controller.(*Controller).startSTSAPIServer(0xc00047ec80, {0x2946a58, 0xc0005443c0}, 0xc00054c700)
        github.com/minio/operator/pkg/controller/main-controller.go:401 +0xa5
created by github.com/minio/operator/pkg/controller.(*Controller).Start in goroutine 1
        github.com/minio/operator/pkg/controller/main-controller.go:535 +0x4bb
root@onprem-node245:/home/onprem_shell#

minio operator csr:

root@onprem-node245:/home/onprem_shell# kubectl get csr -n minio-operator
NAME                     AGE   SIGNERNAME                      REQUESTOR                                             REQUESTEDDURATION   CONDITION
sts-minio-operator-csr   46m   kubernetes.io/kubelet-serving   system:serviceaccount:minio-operator:minio-operator   <none>              Approved,Issued
root@onprem-node245:/home/onprem_shell# kubectl get csr -n minio-operator -o yaml
apiVersion: v1
items:
- apiVersion: certificates.k8s.io/v1
  kind: CertificateSigningRequest
  metadata:
    creationTimestamp: "2025-05-30T03:27:40Z"
    name: sts-minio-operator-csr
    resourceVersion: "21390"
    uid: 8b20ffbd-59ed-4f77-8a1d-a71e87a2f362
  spec:
    extra:
      authentication.kubernetes.io/credential-id:
      - JTI=c6b78449-f259-41ab-96c1-5421b04d5fc2
      authentication.kubernetes.io/node-name:
      - onprem-node245.hstlabs.glcp.hpecorp.net
      authentication.kubernetes.io/node-uid:
      - 6da906c4-9605-4526-a489-6411506de81b
      authentication.kubernetes.io/pod-name:
      - minio-operator-c8b489f59-v8ts6
      authentication.kubernetes.io/pod-uid:
      - 607bded2-428f-42c0-9f28-84b7602b719f
    groups:
    - system:serviceaccounts
    - system:serviceaccounts:minio-operator
    - system:authenticated
    request: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQlh6Q0NBUVVDQVFBd1JERVZNQk1HQTFVRUNoTU1jM2x6ZEdWdE9tNXZaR1Z6TVNzd0tRWURWUVFERXlKegplWE4wWlcwNmJtOWtaVHB6ZEhNdWJXbHVhVzh0YjNCbGNtRjBiM0l1YzNaak1Ga3dFd1lIS29aSXpqMENBUVlJCktvWkl6ajBEQVFjRFFnQUVRbDlSN0djeFQzSGRldVpXOUV3OCtOS3BRQ2VaQUdONlo1RmxaRkZKNlJwSXJOdHUKSjR4aWYrWkp0ZVlwTGRBYmNDdlNCUHhEUTRPSkg2K09ZdXc0UzZCZk1GMEdDU3FHU0liM0RRRUpEakZRTUU0dwpUQVlEVlIwUkJFVXdRNElEYzNSemdoWnpkSE11YldsdWFXOHRiM0JsY21GMGIzSXVjM1pqZ2lSemRITXViV2x1CmFXOHRiM0JsY21GMGIzSXVjM1pqTG1Oc2RYTjBaWEl1Ykc5allXd3dDZ1lJS29aSXpqMEVBd1FEU0FBd1JRSWgKQUpXMEdhUWpNbGJIMDhlem8wakMxNkJNTzFkdkYwTFZqVnNYNTc1U000dDlBaUJrRDlrem5zcUFMVWFZaVRxMQp1WCtCRm5SMTBycVdIQ01DYkhXRlR2b1JGdz09Ci0tLS0tRU5EIENFUlRJRklDQVRFIFJFUVVFU1QtLS0tLQo=
    signerName: kubernetes.io/kubelet-serving
    uid: 22210534-b029-4eee-a994-e6ba2638a71a
    usages:
    - digital signature
    - key encipherment
    - server auth
    username: system:serviceaccount:minio-operator:minio-operator
  status:
    certificate: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN4akNDQWE2Z0F3SUJBZ0lSQUl2MTRYMEFSanJiYWlNZG9kR0NDOE13RFFZSktvWklodmNOQVFFTEJRQXcKSERFYU1CZ0dBMVVFQXd3UlUybHNkbVZ5WTNKbFpXdE5hemh6UTBFd0hoY05NalV3TlRNd01ETXlNalF3V2hjTgpNall3TlRBNE1EY3hOakUyV2pCRU1SVXdFd1lEVlFRS0V3eHplWE4wWlcwNmJtOWtaWE14S3pBcEJnTlZCQU1UCkluTjVjM1JsYlRwdWIyUmxPbk4wY3k1dGFXNXBieTF2Y0dWeVlYUnZjaTV6ZG1Nd1dUQVRCZ2NxaGtqT1BRSUIKQmdncWhrak9QUU1CQndOQ0FBUkNYMUhzWnpGUGNkMTY1bGIwVER6NDBxbEFKNWtBWTNwbmtXVmtVVW5wR2tpcwoyMjRuakdKLzVrbTE1aWt0MEJ0d0s5SUUvRU5EZzRrZnI0NWk3RGhMbzRHbE1JR2lNQTRHQTFVZER3RUIvd1FFCkF3SUZvREFUQmdOVkhTVUVEREFLQmdnckJnRUZCUWNEQVRBTUJnTlZIUk1CQWY4RUFqQUFNQjhHQTFVZEl3UVkKTUJhQUZJQUdtZjUwUGM4QXl2ZmNqanA5TGpsY1FHcDlNRXdHQTFVZEVRUkZNRU9DQTNOMGM0SVdjM1J6TG0xcApibWx2TFc5d1pYSmhkRzl5TG5OMlk0SWtjM1J6TG0xcGJtbHZMVzl3WlhKaGRHOXlMbk4yWXk1amJIVnpkR1Z5CkxteHZZMkZzTUEwR0NTcUdTSWIzRFFFQkN3VUFBNElCQVFBOEFheW02QzkzNEg2Vk00c2IyaUdscGkzU3dscUQKTVdvdTl5Ym5CNld2ZXZFekx1ZzJPSFl3WUFObGxNQ3VJbFp2WEdFL0NoRXd0NEFSaU9vRVhISytmcVJHcHBjYQpGNXNpeG9laU0wZ29nVXBNelovRjZtViszaHIrVVZIaWgxZG5HSHJ4cnZnZ1NqYnJlSkZvOHRIVjFUaWpxYmVHCkI1dk5LVi9Sbk9Oa2wwbFZCTjZXZ25UMG5rei9yL1EyZHlUb1puUXk3WE1RaWNIcWh0THpIQlhFb1lMNmFUbisKM1VIK0owS3R2L1phckVUYWhXRTcyb0J2RmNSMjlRU2crdEg0NDVWeitydXNacHIzTHR6NzhiVkgrN1JEMHRWeApET0xwVmNFMHNtKzVIMzZpUUlXZTcwWm5odEFzUzFmc1NtQmhpSmN6Y3hZbWdxSGo1M3ZzRDlZRQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
    conditions:
    - lastTransitionTime: "2025-05-30T03:27:40Z"
      lastUpdateTime: "2025-05-30T03:27:40Z"
      message: Automatically approved by MinIO Operator
      reason: MinIOOperatorAutoApproval
      status: "True"
      type: Approved
kind: List
metadata:
  resourceVersion: ""

[L1ghtman2k]

Can you post fulll minio-operator logging and run kubectl get csr -n <ns> and post the output? A mismatch between private and public key only occurs when the CSR is created using a different key-pair. This can only happen if no certificate is issued for the private key and the public/private keys get out of sync.

I think I see how problem could occur:

We always re-generate the private key here:

operator/pkg/controller/tls.go

Line 306 in 1e792f1

privateKey, err := newPrivateKey(miniov2.DefaultEllipticCurve)

Then we expect to find the associated public key on this line:

operator/pkg/controller/tls.go

Line 229 in 1e792f1

certBytes, err := c.fetchCertificate(ctx, csrName)

However, what happens if K8s api doesn't generate the Certificate in time? For example, one way this could happen, is if k8s api takes >5 seconds to generate the certificate based on csr: https://github.com/minio/operator/blob/master/pkg/controller/csr.go#L204

Then the code would go back in the loop, and wait 10 seconds before attempting to redo the process:

operator/pkg/controller/tls.go

Line 131 in 1e792f1

time.Sleep(time.Second * 10)

What could happen, is that the CSR finally gets to generate the Certificate in those 10 seconds, leading to private key getting re-generated, but old CSR remaining: https://github.com/minio/operator/blob/master/pkg/controller/csr.go#L105.

Couple of potential solutions:

• increase the timer between approval -> approval,issued from 5 seconds to something larger
• cleanup old CSR after generating new private key

[ramondeklein]

This is actually a bug in the operator. If CSR signing takes too long, then you should delete the secret so it will recreate everything. That's the simplest workaround.