From 7ec0f4126deb8308209e10670ebaa256a3bc3909 Mon Sep 17 00:00:00 2001 From: Nick Larsen Date: Thu, 18 Dec 2025 12:06:25 +0100 Subject: [PATCH 1/3] Add guide for deploying operators and CSI drivers on separate nodes NOTE: This is work-in-progress --- modules/guides/nav.adoc | 1 + ...-operators-and-csi-drivers-separately.adoc | 353 ++++++++++++++++++ 2 files changed, 354 insertions(+) create mode 100644 modules/guides/pages/deploy-operators-and-csi-drivers-separately.adoc diff --git a/modules/guides/nav.adoc b/modules/guides/nav.adoc index bcf28f170..fe595dd2a 100644 --- a/modules/guides/nav.adoc +++ b/modules/guides/nav.adoc @@ -6,3 +6,4 @@ ** xref:viewing-and-verifying-sboms.adoc[] ** xref:enabling-verification-of-image-signatures.adoc[] ** xref:kubernetes-cluster-domain.adoc[] +** xref:deploy-operators-and-csi-drivers-separately.adoc[] diff --git a/modules/guides/pages/deploy-operators-and-csi-drivers-separately.adoc b/modules/guides/pages/deploy-operators-and-csi-drivers-separately.adoc new file mode 100644 index 000000000..80a40c2b7 --- /dev/null +++ b/modules/guides/pages/deploy-operators-and-csi-drivers-separately.adoc @@ -0,0 +1,353 @@ += Deploying operators and CSI drivers on separate nodes +:related-issue: https://github.com/stackabletech/issues/issues/763 +:secret-operator-values: https://github.com/stackabletech/secret-operator/blob/main/deploy/helm/secret-operator/values.yaml +:listener-operator-values: https://github.com/stackabletech/listener-operator/blob/main/deploy/helm/listener-operator/values.yaml +:commons-operator-values: https://github.com/stackabletech/commons-operator/blob/main/deploy/helm/commons-operator/values.yaml +:nifi-operator-values: https://github.com/stackabletech/nifi-operator/blob/main/deploy/helm/nifi-operator/values.yaml +// TODO: Is there a way to make the links above go to the right place? +// Eg: we can search/replace "0.0.0-dev" with the release, but in the above case we would need to replace "main". + +Operators can be installed on nodes separate from where the workloads will run. +There is a caveat when it comes to two operators: Secret Operator and Listener Operator. +They make use of the Container Storage Interface (CSI) and have components that must run on nodes with workloads that mount CSI volumes. + +This guide will show how to schedule operators on one group of nodes (for example, a Karpenter NodePool), while scheduling applicable components where workload will run. + +== Setup + +You will need a Kubernetes cluster with multiple nodes split into two groups: + +* stackable-operators +* stackable-workloads + +TODO: tabs for kind or Karpenter NodePools + +[NOTE] +==== +This guide will use _KinD_ to demonstrate, but if you are using Karpenter (eg: AWK EKS), +you can adjust the labels to be based on the name of your NodePools. + +For example: + +* `karpenter.sh/nodepool: stackable-operators` +* `karpenter.sh/nodepool: stackable-workloads` + +==== + +Create a KinD config called `kind-config.yaml` containing: + +[source,yaml] +---- +kind: Cluster +apiVersion: kind.x-k8s.io/v1alpha4 +nodes: +- role: control-plane +- role: worker + labels: + nodepool: stackable-operators +- role: worker + labels: + nodepool: stackable-operators +- role: worker + labels: + nodepool: stackable-workloads +- role: worker + labels: + nodepool: stackable-workloads +---- + +Launch the cluster: + +[source,bash] +---- +kind create cluster --name stackable --config kind-config.yaml +---- + +You can see which nodes are in which _nodepool_ by using the following command: + +[source,bash] +---- +kubectl get nodes -o json | jq ' + .items[] + | .metadata.name as $name + | .metadata.labels["nodepool"] as $nodepool + | $nodepool//empty + | {"nodename": $name, "nodepool": $nodepool} +' +---- + +== Prepare Helm Values for the Stackable Operators + +[NOTE] +==== +Most Stackable operators use the same Helm Values structure, however Secret and +Listener operator differ slightly - which is what allows the components to be +configured independently of each other. +==== + +// TODO: Move these into files and include them (so we can run them easily) + +[tabs] +==== +Secret Operator:: ++ +-- +Store the values in a file called `stackable-secret-operator.yaml`. + +// TODO: Link to default values + +[source,yaml] +---- +controllerService: + nodeSelector: + nodepool: stackable-operators + +csiNodeDriver: + # Node Drivers need to run on the same nodes as the workloads using them + nodeSelector: + nodepool: stackable-workloads +---- + +-- + +Listener Operator:: ++ +-- +Store the values in a file called `stackable-listener-operator.yaml`. + +// TODO: Link to default values + +[source,yaml] +---- +csiProvisioner: + nodeSelector: + nodepool: stackable-operators + +csiNodeDriver: + # Node Drivers need to run on the same nodes as the workloads using them + nodeSelector: + nodepool: stackable-workloads +---- + +-- + +Remaining operators:: ++ +-- +Store the values in a file called `stackable-operators.yaml`. + +// TODO: Link to default values for remaining operators used in this guide + +[source,yaml] +---- +nodeSelector: + nodepool: stackable-operators +---- + +-- +==== + +NOTE: If you would like to run on nodes with taints, you can list `tolerations` next to the `nodeSelector`. + +== Install the Stackable Operators + +Now install the operators to the applicable node pools by using the Helm overrides + +[tabs] +==== +Secret Operator:: ++ +-- +NOTE: This operator uses a specific values file. + +[source,bash] +---- +helm install secret-operator \ + --version=0.0.0-dev \ + --values=stackable-secret-operator.yaml \ + oci://oci.stackable.tech/sdp-charts/secret-operator +---- + +-- + +Listener Operator:: ++ +-- +NOTE: This operator uses a specific values file. + +[source,bash] +---- +helm install listener-operator \ + --version=0.0.0-dev \ + --values=stackable-listener-operator.yaml \ + oci://oci.stackable.tech/sdp-charts/listener-operator +---- + +-- + +Remaining operators:: ++ +-- +NOTE: These operator use the same values file. + +[source,bash] +---- +helm install commons-operator \ + --version=0.0.0-dev \ + --values=stackable-operators.yaml \ + oci://oci.stackable.tech/sdp-charts/commons-operator + +helm install nifi-operator \ + --version=0.0.0-dev \ + --values=stackable-operators.yaml \ + oci://oci.stackable.tech/sdp-charts/nifi-operator +---- + +-- +==== + +You should now see that the operators are running on the `stackable-operator` nodes, while the CSI drivers are running on the `stackable-workload` nodes. + +Pods running on the `stackable-operators` node pool: + +[source,bash] +---- +OPERATORS_NODEPOOL=$(kubectl get nodes -l nodepool=stackable-operators -o jsonpath="{.items[*].metadata.name}" | tr ' ' ',') +echo "Nodes in operators pool: $OPERATORS_NODEPOOL\n" +kubectl get pods -o json | jq --raw-output --arg nodepool "$OPERATORS_NODEPOOL" '.items[] | .metadata.name as $podname | .spec.nodeName as $nodename | select($nodename | IN($nodepool | split(",")[])) | $podname' +---- + +You should see similar output showing the Stackable Operators are running only on nodes with the label `nodepool: stackable-operators`. + +[source] +--- +Nodes in operators pool: stackable-worker,stackable-worker2 + +commons-operator-deployment-674c469b47-nm5vb +listener-operator-csi-provisioner-85b686d48-hv5kf +nifi-operator-deployment-7c59778bb8-r26b8 +secret-operator-66b85c669d-7hsxs +--- + +Pods running on the `stackable-workloads` node pool: + +[source,bash] +---- +WORKLOADS_NODEPOOL=$(kubectl get nodes -l nodepool=stackable-workloads -o jsonpath="{.items[*].metadata.name}" | tr ' ' ',') +echo "Nodes in workloads pool: $WORKLOADS_NODEPOOL\n" +kubectl get pods -o json | jq --raw-output --arg nodepool "$WORKLOADS_NODEPOOL" '.items[] | .metadata.name as $podname | .spec.nodeName as $nodename | select($nodename | IN($nodepool | split(",")[])) | $podname' +---- + +You should see similar output showing the Stackable CSI Node Drivers are running only on nodes with the label `nodepool: stackable-workloads`. + +[source] +--- +Nodes in workloads pool: stackable-worker3,stackable-worker4 + +listener-operator-csi-node-driver-lv5r4 +listener-operator-csi-node-driver-vdzsq +secret-operator-csi-node-driver-d8sqw +secret-operator-csi-node-driver-zkrv6 +--- + +The CSI Node Drivers register as such. +This can be seen with the driver count being 2 (one for listener-operator volumes, and one for secret-operator volumes) for nodes in the workloads pool: + +[source,console] +---- +$ kubectl get csinodes +NAME DRIVERS AGE +stackable-control-plane 0 3h40m +stackable-worker 0 3h39m +stackable-worker2 0 3h39m +stackable-worker3 2 3h39m +stackable-worker4 2 3h39m +---- + +== Install a workload + +We'll install a NiFi cluster onto a `stackable-workload` node. Create a new file called `nifi.yaml` with the following contents: +// This is taken from the NiFi Getting Started, but with some modifications. +// TODO: Update nifi getting started to remove zookeeper. + +[source,yaml] +---- +--- +apiVersion: v1 +kind: Secret +metadata: + name: simple-admin-credentials +stringData: + admin: admin +--- +apiVersion: authentication.stackable.tech/v1alpha1 +kind: AuthenticationClass +metadata: + name: simple-nifi-users +spec: + provider: + static: + userCredentialsSecret: + name: simple-admin-credentials +--- +apiVersion: nifi.stackable.tech/v1alpha1 +kind: NifiCluster +metadata: + name: simple-nifi +spec: + image: + productVersion: 2.6.0 + clusterConfig: + authentication: + - authenticationClass: simple-nifi-users + sensitiveProperties: + keySecret: nifi-sensitive-property-key + autoGenerate: true + nodes: + roleGroups: + default: + replicas: 1 + config: + # Run NiFi nodes in the workloads pool + affinity: + nodeSelector: + nodepool: stackable-workloads +---- + +Apply it to Kubernetes: + +[source,console] +---- +$ kubectl apply -f nifi.yaml +---- + +Then take a look at the pods running on nodes with the label `nodepool: stackable-workloads`: + +[source,bash] +---- +WORKLOADS_NODEPOOL=$(kubectl get nodes -l nodepool=stackable-workloads -o jsonpath="{.items[*].metadata.name}" | tr ' ' ',') +echo "Nodes in workloads pool: $WORKLOADS_NODEPOOL\n" +kubectl get pods -o json | jq --raw-output --arg nodepool "$WORKLOADS_NODEPOOL" '.items[] | .metadata.name as $podname | .spec.nodeName as $nodename | select($nodename | IN($nodepool | split(",")[])) | $podname' +---- + +You should see similar output as last time, but now with the NiFi pod. + +[source] +--- +Nodes in workloads pool: stackable-worker3,stackable-worker4 + +listener-operator-csi-node-driver-lv5r4 +listener-operator-csi-node-driver-vdzsq +secret-operator-csi-node-driver-d8sqw +secret-operator-csi-node-driver-zkrv6 +simple-nifi-node-default-0 +--- + +== Cleanup + +Once done, you can delete the KinD cluster like so: + +[source,bash] +---- +kind delete cluster --name stackable +---- From 01da299e60f6245940f56a73fe971886c1d3dd46 Mon Sep 17 00:00:00 2001 From: Nick Larsen Date: Fri, 19 Dec 2025 16:47:25 +0100 Subject: [PATCH 2/3] wip --- .../pages/deploy-operators-and-csi-drivers-separately.adoc | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/modules/guides/pages/deploy-operators-and-csi-drivers-separately.adoc b/modules/guides/pages/deploy-operators-and-csi-drivers-separately.adoc index 80a40c2b7..a77e5575d 100644 --- a/modules/guides/pages/deploy-operators-and-csi-drivers-separately.adoc +++ b/modules/guides/pages/deploy-operators-and-csi-drivers-separately.adoc @@ -20,7 +20,7 @@ You will need a Kubernetes cluster with multiple nodes split into two groups: * stackable-operators * stackable-workloads -TODO: tabs for kind or Karpenter NodePools +// TODO: Add an image to illustrate what the guide aims to achieve [NOTE] ==== @@ -333,7 +333,7 @@ kubectl get pods -o json | jq --raw-output --arg nodepool "$WORKLOADS_NODEPOOL" You should see similar output as last time, but now with the NiFi pod. [source] ---- +---- Nodes in workloads pool: stackable-worker3,stackable-worker4 listener-operator-csi-node-driver-lv5r4 @@ -341,7 +341,7 @@ listener-operator-csi-node-driver-vdzsq secret-operator-csi-node-driver-d8sqw secret-operator-csi-node-driver-zkrv6 simple-nifi-node-default-0 ---- +---- == Cleanup From 5481429900a144c30c9a5aa850d553af68ddda12 Mon Sep 17 00:00:00 2001 From: Nick Larsen Date: Fri, 19 Dec 2025 16:48:21 +0100 Subject: [PATCH 3/3] wip --- .../deploy-operators-and-csi-drivers-separately.adoc | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/modules/guides/pages/deploy-operators-and-csi-drivers-separately.adoc b/modules/guides/pages/deploy-operators-and-csi-drivers-separately.adoc index a77e5575d..f17ec3375 100644 --- a/modules/guides/pages/deploy-operators-and-csi-drivers-separately.adoc +++ b/modules/guides/pages/deploy-operators-and-csi-drivers-separately.adoc @@ -220,14 +220,14 @@ kubectl get pods -o json | jq --raw-output --arg nodepool "$OPERATORS_NODEPOOL" You should see similar output showing the Stackable Operators are running only on nodes with the label `nodepool: stackable-operators`. [source] ---- +---- Nodes in operators pool: stackable-worker,stackable-worker2 commons-operator-deployment-674c469b47-nm5vb listener-operator-csi-provisioner-85b686d48-hv5kf nifi-operator-deployment-7c59778bb8-r26b8 secret-operator-66b85c669d-7hsxs ---- +---- Pods running on the `stackable-workloads` node pool: @@ -241,14 +241,14 @@ kubectl get pods -o json | jq --raw-output --arg nodepool "$WORKLOADS_NODEPOOL" You should see similar output showing the Stackable CSI Node Drivers are running only on nodes with the label `nodepool: stackable-workloads`. [source] ---- +---- Nodes in workloads pool: stackable-worker3,stackable-worker4 listener-operator-csi-node-driver-lv5r4 listener-operator-csi-node-driver-vdzsq secret-operator-csi-node-driver-d8sqw secret-operator-csi-node-driver-zkrv6 ---- +---- The CSI Node Drivers register as such. This can be seen with the driver count being 2 (one for listener-operator volumes, and one for secret-operator volumes) for nodes in the workloads pool: