diff --git a/modules/guides/nav.adoc b/modules/guides/nav.adoc index bcf28f170..fe595dd2a 100644 --- a/modules/guides/nav.adoc +++ b/modules/guides/nav.adoc @@ -6,3 +6,4 @@ ** xref:viewing-and-verifying-sboms.adoc[] ** xref:enabling-verification-of-image-signatures.adoc[] ** xref:kubernetes-cluster-domain.adoc[] +** xref:deploy-operators-and-csi-drivers-separately.adoc[] diff --git a/modules/guides/pages/deploy-operators-and-csi-drivers-separately.adoc b/modules/guides/pages/deploy-operators-and-csi-drivers-separately.adoc new file mode 100644 index 000000000..f17ec3375 --- /dev/null +++ b/modules/guides/pages/deploy-operators-and-csi-drivers-separately.adoc @@ -0,0 +1,353 @@ += Deploying operators and CSI drivers on separate nodes +:related-issue: https://github.com/stackabletech/issues/issues/763 +:secret-operator-values: https://github.com/stackabletech/secret-operator/blob/main/deploy/helm/secret-operator/values.yaml +:listener-operator-values: https://github.com/stackabletech/listener-operator/blob/main/deploy/helm/listener-operator/values.yaml +:commons-operator-values: https://github.com/stackabletech/commons-operator/blob/main/deploy/helm/commons-operator/values.yaml +:nifi-operator-values: https://github.com/stackabletech/nifi-operator/blob/main/deploy/helm/nifi-operator/values.yaml +// TODO: Is there a way to make the links above go to the right place? +// Eg: we can search/replace "0.0.0-dev" with the release, but in the above case we would need to replace "main". + +Operators can be installed on nodes separate from where the workloads will run. +There is a caveat when it comes to two operators: Secret Operator and Listener Operator. +They make use of the Container Storage Interface (CSI) and have components that must run on nodes with workloads that mount CSI volumes. + +This guide will show how to schedule operators on one group of nodes (for example, a Karpenter NodePool), while scheduling applicable components where workload will run. + +== Setup + +You will need a Kubernetes cluster with multiple nodes split into two groups: + +* stackable-operators +* stackable-workloads + +// TODO: Add an image to illustrate what the guide aims to achieve + +[NOTE] +==== +This guide will use _KinD_ to demonstrate, but if you are using Karpenter (eg: AWK EKS), +you can adjust the labels to be based on the name of your NodePools. + +For example: + +* `karpenter.sh/nodepool: stackable-operators` +* `karpenter.sh/nodepool: stackable-workloads` + +==== + +Create a KinD config called `kind-config.yaml` containing: + +[source,yaml] +---- +kind: Cluster +apiVersion: kind.x-k8s.io/v1alpha4 +nodes: +- role: control-plane +- role: worker + labels: + nodepool: stackable-operators +- role: worker + labels: + nodepool: stackable-operators +- role: worker + labels: + nodepool: stackable-workloads +- role: worker + labels: + nodepool: stackable-workloads +---- + +Launch the cluster: + +[source,bash] +---- +kind create cluster --name stackable --config kind-config.yaml +---- + +You can see which nodes are in which _nodepool_ by using the following command: + +[source,bash] +---- +kubectl get nodes -o json | jq ' + .items[] + | .metadata.name as $name + | .metadata.labels["nodepool"] as $nodepool + | $nodepool//empty + | {"nodename": $name, "nodepool": $nodepool} +' +---- + +== Prepare Helm Values for the Stackable Operators + +[NOTE] +==== +Most Stackable operators use the same Helm Values structure, however Secret and +Listener operator differ slightly - which is what allows the components to be +configured independently of each other. +==== + +// TODO: Move these into files and include them (so we can run them easily) + +[tabs] +==== +Secret Operator:: ++ +-- +Store the values in a file called `stackable-secret-operator.yaml`. + +// TODO: Link to default values + +[source,yaml] +---- +controllerService: + nodeSelector: + nodepool: stackable-operators + +csiNodeDriver: + # Node Drivers need to run on the same nodes as the workloads using them + nodeSelector: + nodepool: stackable-workloads +---- + +-- + +Listener Operator:: ++ +-- +Store the values in a file called `stackable-listener-operator.yaml`. + +// TODO: Link to default values + +[source,yaml] +---- +csiProvisioner: + nodeSelector: + nodepool: stackable-operators + +csiNodeDriver: + # Node Drivers need to run on the same nodes as the workloads using them + nodeSelector: + nodepool: stackable-workloads +---- + +-- + +Remaining operators:: ++ +-- +Store the values in a file called `stackable-operators.yaml`. + +// TODO: Link to default values for remaining operators used in this guide + +[source,yaml] +---- +nodeSelector: + nodepool: stackable-operators +---- + +-- +==== + +NOTE: If you would like to run on nodes with taints, you can list `tolerations` next to the `nodeSelector`. + +== Install the Stackable Operators + +Now install the operators to the applicable node pools by using the Helm overrides + +[tabs] +==== +Secret Operator:: ++ +-- +NOTE: This operator uses a specific values file. + +[source,bash] +---- +helm install secret-operator \ + --version=0.0.0-dev \ + --values=stackable-secret-operator.yaml \ + oci://oci.stackable.tech/sdp-charts/secret-operator +---- + +-- + +Listener Operator:: ++ +-- +NOTE: This operator uses a specific values file. + +[source,bash] +---- +helm install listener-operator \ + --version=0.0.0-dev \ + --values=stackable-listener-operator.yaml \ + oci://oci.stackable.tech/sdp-charts/listener-operator +---- + +-- + +Remaining operators:: ++ +-- +NOTE: These operator use the same values file. + +[source,bash] +---- +helm install commons-operator \ + --version=0.0.0-dev \ + --values=stackable-operators.yaml \ + oci://oci.stackable.tech/sdp-charts/commons-operator + +helm install nifi-operator \ + --version=0.0.0-dev \ + --values=stackable-operators.yaml \ + oci://oci.stackable.tech/sdp-charts/nifi-operator +---- + +-- +==== + +You should now see that the operators are running on the `stackable-operator` nodes, while the CSI drivers are running on the `stackable-workload` nodes. + +Pods running on the `stackable-operators` node pool: + +[source,bash] +---- +OPERATORS_NODEPOOL=$(kubectl get nodes -l nodepool=stackable-operators -o jsonpath="{.items[*].metadata.name}" | tr ' ' ',') +echo "Nodes in operators pool: $OPERATORS_NODEPOOL\n" +kubectl get pods -o json | jq --raw-output --arg nodepool "$OPERATORS_NODEPOOL" '.items[] | .metadata.name as $podname | .spec.nodeName as $nodename | select($nodename | IN($nodepool | split(",")[])) | $podname' +---- + +You should see similar output showing the Stackable Operators are running only on nodes with the label `nodepool: stackable-operators`. + +[source] +---- +Nodes in operators pool: stackable-worker,stackable-worker2 + +commons-operator-deployment-674c469b47-nm5vb +listener-operator-csi-provisioner-85b686d48-hv5kf +nifi-operator-deployment-7c59778bb8-r26b8 +secret-operator-66b85c669d-7hsxs +---- + +Pods running on the `stackable-workloads` node pool: + +[source,bash] +---- +WORKLOADS_NODEPOOL=$(kubectl get nodes -l nodepool=stackable-workloads -o jsonpath="{.items[*].metadata.name}" | tr ' ' ',') +echo "Nodes in workloads pool: $WORKLOADS_NODEPOOL\n" +kubectl get pods -o json | jq --raw-output --arg nodepool "$WORKLOADS_NODEPOOL" '.items[] | .metadata.name as $podname | .spec.nodeName as $nodename | select($nodename | IN($nodepool | split(",")[])) | $podname' +---- + +You should see similar output showing the Stackable CSI Node Drivers are running only on nodes with the label `nodepool: stackable-workloads`. + +[source] +---- +Nodes in workloads pool: stackable-worker3,stackable-worker4 + +listener-operator-csi-node-driver-lv5r4 +listener-operator-csi-node-driver-vdzsq +secret-operator-csi-node-driver-d8sqw +secret-operator-csi-node-driver-zkrv6 +---- + +The CSI Node Drivers register as such. +This can be seen with the driver count being 2 (one for listener-operator volumes, and one for secret-operator volumes) for nodes in the workloads pool: + +[source,console] +---- +$ kubectl get csinodes +NAME DRIVERS AGE +stackable-control-plane 0 3h40m +stackable-worker 0 3h39m +stackable-worker2 0 3h39m +stackable-worker3 2 3h39m +stackable-worker4 2 3h39m +---- + +== Install a workload + +We'll install a NiFi cluster onto a `stackable-workload` node. Create a new file called `nifi.yaml` with the following contents: +// This is taken from the NiFi Getting Started, but with some modifications. +// TODO: Update nifi getting started to remove zookeeper. + +[source,yaml] +---- +--- +apiVersion: v1 +kind: Secret +metadata: + name: simple-admin-credentials +stringData: + admin: admin +--- +apiVersion: authentication.stackable.tech/v1alpha1 +kind: AuthenticationClass +metadata: + name: simple-nifi-users +spec: + provider: + static: + userCredentialsSecret: + name: simple-admin-credentials +--- +apiVersion: nifi.stackable.tech/v1alpha1 +kind: NifiCluster +metadata: + name: simple-nifi +spec: + image: + productVersion: 2.6.0 + clusterConfig: + authentication: + - authenticationClass: simple-nifi-users + sensitiveProperties: + keySecret: nifi-sensitive-property-key + autoGenerate: true + nodes: + roleGroups: + default: + replicas: 1 + config: + # Run NiFi nodes in the workloads pool + affinity: + nodeSelector: + nodepool: stackable-workloads +---- + +Apply it to Kubernetes: + +[source,console] +---- +$ kubectl apply -f nifi.yaml +---- + +Then take a look at the pods running on nodes with the label `nodepool: stackable-workloads`: + +[source,bash] +---- +WORKLOADS_NODEPOOL=$(kubectl get nodes -l nodepool=stackable-workloads -o jsonpath="{.items[*].metadata.name}" | tr ' ' ',') +echo "Nodes in workloads pool: $WORKLOADS_NODEPOOL\n" +kubectl get pods -o json | jq --raw-output --arg nodepool "$WORKLOADS_NODEPOOL" '.items[] | .metadata.name as $podname | .spec.nodeName as $nodename | select($nodename | IN($nodepool | split(",")[])) | $podname' +---- + +You should see similar output as last time, but now with the NiFi pod. + +[source] +---- +Nodes in workloads pool: stackable-worker3,stackable-worker4 + +listener-operator-csi-node-driver-lv5r4 +listener-operator-csi-node-driver-vdzsq +secret-operator-csi-node-driver-d8sqw +secret-operator-csi-node-driver-zkrv6 +simple-nifi-node-default-0 +---- + +== Cleanup + +Once done, you can delete the KinD cluster like so: + +[source,bash] +---- +kind delete cluster --name stackable +----