From ce0ee3fe61cb3a499a6b1fd5c0cf9246c8a8a1e4 Mon Sep 17 00:00:00 2001 From: Mark Phelps <209477+markphelps@users.noreply.github.com> Date: Wed, 25 Feb 2026 16:23:34 +0000 Subject: [PATCH 1/5] docs: add Kubernetes troubleshooting guide Cover common issues encountered deploying Flipt v2 to Kubernetes: - Helm schema validation errors (--skip-schema-validation) - Machine ID not found on containerd clusters (GKE, EKS) - Persistent volume configuration and storage backend paths - Secrets management with GitOps (ArgoCD, FluxCD) - flipt license check vs Helm values confusion Based on real deployment issues from community support. --- docs/docs.json | 3 +- .../deployment/kubernetes-troubleshooting.mdx | 251 ++++++++++++++++++ 2 files changed, 253 insertions(+), 1 deletion(-) create mode 100644 docs/v2/guides/operations/deployment/kubernetes-troubleshooting.mdx diff --git a/docs/docs.json b/docs/docs.json index c33b225..7af0284 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -128,7 +128,8 @@ { "group": "Deployment", "pages": [ - "v2/guides/operations/deployment/deploy-to-kubernetes" + "v2/guides/operations/deployment/deploy-to-kubernetes", + "v2/guides/operations/deployment/kubernetes-troubleshooting" ] }, { diff --git a/docs/v2/guides/operations/deployment/kubernetes-troubleshooting.mdx b/docs/v2/guides/operations/deployment/kubernetes-troubleshooting.mdx new file mode 100644 index 0000000..e70af57 --- /dev/null +++ b/docs/v2/guides/operations/deployment/kubernetes-troubleshooting.mdx @@ -0,0 +1,251 @@ +--- +title: "Kubernetes Troubleshooting" +description: "Common issues and solutions when deploying Flipt v2 to Kubernetes" +--- + +This guide covers common issues encountered when deploying Flipt v2 to Kubernetes using the official Helm chart, along with their solutions. + +## Helm Schema Validation Errors + +When installing or upgrading the Flipt Helm chart, you may encounter schema validation errors like: + +``` +helm install flipt flipt/flipt-v2 -f values.yaml +Error: INSTALLATION FAILED: values don't meet the specifications of the schema(s) in the following chart(s): +flipt-v2: +- (root): Additional property pgb is not allowed +- flipt.config.storage.bitbucket.backend: Additional property local is not allowed +- autoscaling: Additional property targetMemoryUtilizationPercentage is not allowed +- resources: Additional property limits is not allowed +- resources: Additional property requests is not allowed +``` + +This happens when the chart's JSON schema doesn't cover all valid configuration options. To work around this, skip schema validation: + +```bash +helm install flipt flipt/flipt-v2 -f values.yaml --skip-schema-validation +``` + +Or for upgrades: + +```bash +helm upgrade flipt flipt/flipt-v2 -f values.yaml --skip-schema-validation +``` + + + The Helm chart schema is being updated to cover all valid configurations. In + the meantime, `--skip-schema-validation` is safe to use and does not affect + functionality. + + +## Machine ID Not Found (License Validation) + +When running Flipt Pro on containerd-based Kubernetes clusters (such as GKE or modern EKS), you may see: + +``` +license is invalid; additional features are disabled. +{"error": "machineid: machineid: no machine-id found"} +``` + +### Why This Happens + +Flipt uses a machine fingerprint for license validation. The fingerprinting library reads from: + +1. `/var/lib/dbus/machine-id` +2. `/etc/machine-id` +3. Docker-specific paths in `/proc/self/cgroup` and `/proc/self/mountinfo` + +In containerd-based environments (GKE, EKS with containerd, etc.), none of these files exist inside the container because: + +- The Flipt container image is Alpine-based and doesn't include dbus +- `/etc/machine-id` is not present in minimal container images +- The Docker-specific fallback paths don't match containerd's format + +### Solution: Mount the Host Machine ID + +Mount the Kubernetes node's `/etc/machine-id` into the Flipt container by adding these values to your `values.yaml`: + +```yaml +extraVolumeMounts: + - name: machine-id + mountPath: /etc/machine-id + readOnly: true + +extraVolumes: + - name: machine-id + hostPath: + path: /etc/machine-id +``` + +This mounts the host node's machine ID into the container, allowing the license system to identify the machine. + + + If your cluster's security policy restricts `hostPath` volumes, you may need + to request an exception for this read-only mount, or wait for the configurable + machine fingerprint feature + ([flipt-io/flipt#5426](https://github.com/flipt-io/flipt/issues/5426)). + + +### Affected Environments + +- Google Kubernetes Engine (GKE) +- Amazon EKS with containerd runtime +- Any Kubernetes cluster using containerd instead of Docker +- Rootless container environments +- Serverless container runtimes (AWS Fargate, Google Cloud Run) + +## Persistent Volume Configuration + +Flipt needs a writable directory for local state. If you see read-only filesystem errors or Flipt fails to start with write permission errors, you likely need to configure persistence. + +### Enable Persistence + +Add the following to your `values.yaml`: + +```yaml +persistence: + enabled: true + storageClass: "standard" # Use your cluster's storage class + size: 10Gi +``` + + + On GKE, the default storage class is `standard`. On EKS, it's typically `gp2` + or `gp3`. Run `kubectl get storageclass` to see available options. + + +### Match the Storage Backend Path + +The `path` in your Flipt storage backend configuration refers to the path **inside the container**, not on the host. It must match where the PersistentVolumeClaim is mounted. + +The Helm chart mounts the PVC at `/var/opt/flipt` by default. Your storage backend configuration should use this path: + +```yaml +flipt: + config: + storage: + my-storage: + backend: + type: local + local: + path: /var/opt/flipt +``` + + + Do not use a relative path like `"."` for the backend path. This writes to the + container's working directory, which may be read-only. Always use the absolute + path where the PVC is mounted. + + +## Secrets Management with GitOps + +When deploying Flipt via ArgoCD, FluxCD, or other GitOps tools, you should not commit sensitive values (license keys, Git access tokens) to your Git repository inside `values.yaml`. + +### Using Kubernetes Secrets with Environment Variable Substitution + +Flipt supports [environment variable substitution](/v2/configuration/overview#environment-substitution-and-secret-references) in configuration values using the `${env:VAR_NAME}` syntax. Combined with the Helm chart's `envFrom` support, this lets you keep secrets out of your values file. + +**Step 1:** Create a Kubernetes Secret containing your sensitive values: + +```bash +kubectl create secret generic flipt-secrets \ + --from-literal=FLIPT_LICENSE_KEY=your-license-key \ + --from-literal=FLIPT_GIT_ACCESS_TOKEN=your-access-token +``` + +Or use [External Secrets Operator](https://external-secrets.io/) to sync from your cloud provider's secret manager (GCP Secret Manager, AWS Secrets Manager, Azure Key Vault, HashiCorp Vault). + +**Step 2:** Reference the Secret in your `values.yaml`: + +```yaml +flipt: + envFrom: + - secretRef: + name: flipt-secrets +``` + +**Step 3:** Use environment variable references in your Flipt configuration: + +```yaml +flipt: + config: + license: + key: ${env:FLIPT_LICENSE_KEY} + credentials: + my-git: + type: access_token + access_token: ${env:FLIPT_GIT_ACCESS_TOKEN} + storage: + my-storage: + remote: "https://github.com/your-org/flipt-config.git" + branch: main + credentials: "my-git" +``` + +This approach keeps your `values.yaml` free of secrets and safe to commit to Git. + +### Compatible Secret Management Tools + +This pattern works with any tool that creates Kubernetes Secrets: + +- **[External Secrets Operator](https://external-secrets.io/)** — syncs secrets from GCP Secret Manager, AWS Secrets Manager, Azure Key Vault, HashiCorp Vault +- **[Sealed Secrets](https://sealed-secrets.netlify.app/)** — encrypt secrets client-side, commit encrypted `SealedSecret` resources to Git +- **[ArgoCD Vault Plugin](https://argocd-vault-plugin.readthedocs.io/)** — inject secrets from Vault at deploy time +- **`kubectl create secret`** — create secrets manually for simpler setups + +### Using Individual Environment Variables + +If you prefer to inject secrets as individual environment variables rather than from a Secret reference, use `extraEnvVars`: + +```yaml +flipt: + extraEnvVars: + - name: FLIPT_LICENSE_KEY + valueFrom: + secretKeyRef: + name: flipt-license + key: license-key + - name: FLIPT_GIT_ACCESS_TOKEN + valueFrom: + secretKeyRef: + name: flipt-git-creds + key: access-token +``` + +## flipt license check and Helm Values + +The `flipt license check` command validates your license using a **Flipt configuration file**, not a Helm `values.yaml` file. + +If you run: + +```bash +flipt license check --config values.yaml +``` + +It will not work correctly because the Helm values file wraps the Flipt configuration under `flipt.config`, while `flipt license check` expects the raw Flipt configuration format. + +### Check License Locally + +To check your license locally, create a standalone Flipt configuration file: + +```yaml +# flipt-config.yaml +license: + key: "your-license-key" +``` + +Then run: + +```bash +flipt license check --config flipt-config.yaml +``` + +### Check License in a Running Pod + +To verify the license in your deployed Flipt instance: + +```bash +kubectl exec -it deploy/flipt-v2 -- flipt license check +``` + +This runs the check inside the container where the full configuration and environment variables are available. From 01e00bd7e4402f68b534fe8c3d19cff5de99e34d Mon Sep 17 00:00:00 2001 From: Mark Phelps <209477+markphelps@users.noreply.github.com> Date: Wed, 25 Feb 2026 11:38:48 -0500 Subject: [PATCH 2/5] docs: address review feedback on k8s troubleshooting guide Apply suggested changes from PR review: add missing language tags on code blocks, fix heading casing, remove Cloud Run from affected environments, make schema note evergreen, add Helm version requirement. Also link to the troubleshooting guide from the deploy guide and installation page. Signed-off-by: Mark Phelps <209477+markphelps@users.noreply.github.com> --- .../deployment/deploy-to-kubernetes.mdx | 2 ++ .../deployment/kubernetes-troubleshooting.mdx | 16 ++++++++-------- docs/v2/installation.mdx | 2 +- 3 files changed, 11 insertions(+), 9 deletions(-) diff --git a/docs/v2/guides/operations/deployment/deploy-to-kubernetes.mdx b/docs/v2/guides/operations/deployment/deploy-to-kubernetes.mdx index fd75520..d326a8b 100644 --- a/docs/v2/guides/operations/deployment/deploy-to-kubernetes.mdx +++ b/docs/v2/guides/operations/deployment/deploy-to-kubernetes.mdx @@ -202,6 +202,7 @@ You should be able to take the knowledge you've gained in this guide and deploy - [v2 Configuration Documentation](/v2/configuration/overview) - Complete configuration reference - [Git Storage Guide](/v2/guides/operations/environments/git-sync) - Detailed Git storage setup - [Authentication Methods](/v2/configuration/authentication) - Setting up authentication in v2 +- [Kubernetes Troubleshooting](/v2/guides/operations/deployment/kubernetes-troubleshooting) - Common issues and solutions for Kubernetes deployments ### Production Considerations @@ -213,3 +214,4 @@ For production deployments, consider: - **Security**: Enable authentication and configure RBAC policies - **Monitoring**: Set up observability with metrics and distributed tracing - **Backup**: Implement backup strategies for your Git repositories and any local data +- **Troubleshooting**: See the [Kubernetes Troubleshooting](/v2/guides/operations/deployment/kubernetes-troubleshooting) guide for common deployment issues diff --git a/docs/v2/guides/operations/deployment/kubernetes-troubleshooting.mdx b/docs/v2/guides/operations/deployment/kubernetes-troubleshooting.mdx index e70af57..6f8ac76 100644 --- a/docs/v2/guides/operations/deployment/kubernetes-troubleshooting.mdx +++ b/docs/v2/guides/operations/deployment/kubernetes-troubleshooting.mdx @@ -9,7 +9,7 @@ This guide covers common issues encountered when deploying Flipt v2 to Kubernete When installing or upgrading the Flipt Helm chart, you may encounter schema validation errors like: -``` +```text helm install flipt flipt/flipt-v2 -f values.yaml Error: INSTALLATION FAILED: values don't meet the specifications of the schema(s) in the following chart(s): flipt-v2: @@ -20,7 +20,7 @@ flipt-v2: - resources: Additional property requests is not allowed ``` -This happens when the chart's JSON schema doesn't cover all valid configuration options. To work around this, skip schema validation: +This happens when the chart's JSON schema doesn't cover all valid configuration options. To work around this, skip schema validation (requires Helm 3.13+): ```bash helm install flipt flipt/flipt-v2 -f values.yaml --skip-schema-validation @@ -33,16 +33,16 @@ helm upgrade flipt flipt/flipt-v2 -f values.yaml --skip-schema-validation ``` - The Helm chart schema is being updated to cover all valid configurations. In - the meantime, `--skip-schema-validation` is safe to use and does not affect - functionality. + The `--skip-schema-validation` flag is safe to use and does not affect + functionality. Track schema updates in + [flipt-io/helm-charts](https://github.com/flipt-io/helm-charts). ## Machine ID Not Found (License Validation) When running Flipt Pro on containerd-based Kubernetes clusters (such as GKE or modern EKS), you may see: -``` +```text license is invalid; additional features are disabled. {"error": "machineid: machineid: no machine-id found"} ``` @@ -92,7 +92,7 @@ This mounts the host node's machine ID into the container, allowing the license - Amazon EKS with containerd runtime - Any Kubernetes cluster using containerd instead of Docker - Rootless container environments -- Serverless container runtimes (AWS Fargate, Google Cloud Run) +- Amazon EKS with AWS Fargate node groups ## Persistent Volume Configuration @@ -212,7 +212,7 @@ flipt: key: access-token ``` -## flipt license check and Helm Values +## `flipt license check` and Helm Values The `flipt license check` command validates your license using a **Flipt configuration file**, not a Helm `values.yaml` file. diff --git a/docs/v2/installation.mdx b/docs/v2/installation.mdx index b43c3bd..df1ecc3 100644 --- a/docs/v2/installation.mdx +++ b/docs/v2/installation.mdx @@ -270,4 +270,4 @@ flipt: guide](/v2/guides/operations/environments/git-sync). -For comprehensive deployment instructions and configuration examples, see the [Deploy Flipt v2 to Kubernetes](/v2/guides/operations/deployment/deploy-to-kubernetes) guide. +For comprehensive deployment instructions and configuration examples, see the [Deploy Flipt v2 to Kubernetes](/v2/guides/operations/deployment/deploy-to-kubernetes) guide. If you run into issues, check the [Kubernetes Troubleshooting](/v2/guides/operations/deployment/kubernetes-troubleshooting) guide. From 7bf6e8072c1ef506ea3d3795c7051adfb33628cc Mon Sep 17 00:00:00 2001 From: Mark Phelps <209477+markphelps@users.noreply.github.com> Date: Wed, 25 Feb 2026 13:23:00 -0500 Subject: [PATCH 3/5] docs: address second round of review feedback on k8s troubleshooting - Reword --skip-schema-validation note to clarify it bypasses all checks - Use descriptive link text for GitHub issue reference - Drop platform-specific storage class assertions in favor of kubectl - Add release name comment to kubectl exec example - Remove duplicate troubleshooting link from Production Considerations Signed-off-by: Mark Phelps <209477+markphelps@users.noreply.github.com> --- .../deployment/deploy-to-kubernetes.mdx | 1 - .../deployment/kubernetes-troubleshooting.mdx | 19 +++++++++++-------- 2 files changed, 11 insertions(+), 9 deletions(-) diff --git a/docs/v2/guides/operations/deployment/deploy-to-kubernetes.mdx b/docs/v2/guides/operations/deployment/deploy-to-kubernetes.mdx index d326a8b..2a8cd30 100644 --- a/docs/v2/guides/operations/deployment/deploy-to-kubernetes.mdx +++ b/docs/v2/guides/operations/deployment/deploy-to-kubernetes.mdx @@ -214,4 +214,3 @@ For production deployments, consider: - **Security**: Enable authentication and configure RBAC policies - **Monitoring**: Set up observability with metrics and distributed tracing - **Backup**: Implement backup strategies for your Git repositories and any local data -- **Troubleshooting**: See the [Kubernetes Troubleshooting](/v2/guides/operations/deployment/kubernetes-troubleshooting) guide for common deployment issues diff --git a/docs/v2/guides/operations/deployment/kubernetes-troubleshooting.mdx b/docs/v2/guides/operations/deployment/kubernetes-troubleshooting.mdx index 6f8ac76..49fb872 100644 --- a/docs/v2/guides/operations/deployment/kubernetes-troubleshooting.mdx +++ b/docs/v2/guides/operations/deployment/kubernetes-troubleshooting.mdx @@ -33,9 +33,11 @@ helm upgrade flipt flipt/flipt-v2 -f values.yaml --skip-schema-validation ``` - The `--skip-schema-validation` flag is safe to use and does not affect - functionality. Track schema updates in - [flipt-io/helm-charts](https://github.com/flipt-io/helm-charts). + `--skip-schema-validation` bypasses chart schema checks entirely. Use it as a + temporary workaround while tracking upstream schema fixes in + [flipt-io/helm-charts](https://github.com/flipt-io/helm-charts) — typos in + your `values.yaml` will no longer be caught automatically when this flag is + set. ## Machine ID Not Found (License Validation) @@ -81,9 +83,9 @@ This mounts the host node's machine ID into the container, allowing the license If your cluster's security policy restricts `hostPath` volumes, you may need - to request an exception for this read-only mount, or wait for the configurable - machine fingerprint feature - ([flipt-io/flipt#5426](https://github.com/flipt-io/flipt/issues/5426)). + to request an exception for this read-only mount, or wait for the + [configurable machine fingerprint feature](https://github.com/flipt-io/flipt/issues/5426) + to ship. ### Affected Environments @@ -110,8 +112,8 @@ persistence: ``` - On GKE, the default storage class is `standard`. On EKS, it's typically `gp2` - or `gp3`. Run `kubectl get storageclass` to see available options. + Run `kubectl get storageclass` to see available storage classes in your + cluster. The default varies by provider and cluster configuration. ### Match the Storage Backend Path @@ -245,6 +247,7 @@ flipt license check --config flipt-config.yaml To verify the license in your deployed Flipt instance: ```bash +# Replace flipt-v2 with your Helm release name kubectl exec -it deploy/flipt-v2 -- flipt license check ``` From b8586a081354f0da600a72203bcb2f1e9d0e24da Mon Sep 17 00:00:00 2001 From: Mark Phelps <209477+markphelps@users.noreply.github.com> Date: Wed, 25 Feb 2026 20:24:28 +0000 Subject: [PATCH 4/5] docs: remove flipt license check section from troubleshooting --- .../deployment/kubernetes-troubleshooting.mdx | 43 +------------------ 1 file changed, 2 insertions(+), 41 deletions(-) diff --git a/docs/v2/guides/operations/deployment/kubernetes-troubleshooting.mdx b/docs/v2/guides/operations/deployment/kubernetes-troubleshooting.mdx index 49fb872..1bebe1e 100644 --- a/docs/v2/guides/operations/deployment/kubernetes-troubleshooting.mdx +++ b/docs/v2/guides/operations/deployment/kubernetes-troubleshooting.mdx @@ -84,8 +84,8 @@ This mounts the host node's machine ID into the container, allowing the license If your cluster's security policy restricts `hostPath` volumes, you may need to request an exception for this read-only mount, or wait for the - [configurable machine fingerprint feature](https://github.com/flipt-io/flipt/issues/5426) - to ship. + [configurable machine fingerprint + feature](https://github.com/flipt-io/flipt/issues/5426) to ship. ### Affected Environments @@ -213,42 +213,3 @@ flipt: name: flipt-git-creds key: access-token ``` - -## `flipt license check` and Helm Values - -The `flipt license check` command validates your license using a **Flipt configuration file**, not a Helm `values.yaml` file. - -If you run: - -```bash -flipt license check --config values.yaml -``` - -It will not work correctly because the Helm values file wraps the Flipt configuration under `flipt.config`, while `flipt license check` expects the raw Flipt configuration format. - -### Check License Locally - -To check your license locally, create a standalone Flipt configuration file: - -```yaml -# flipt-config.yaml -license: - key: "your-license-key" -``` - -Then run: - -```bash -flipt license check --config flipt-config.yaml -``` - -### Check License in a Running Pod - -To verify the license in your deployed Flipt instance: - -```bash -# Replace flipt-v2 with your Helm release name -kubectl exec -it deploy/flipt-v2 -- flipt license check -``` - -This runs the check inside the container where the full configuration and environment variables are available. From 0c84ca51fd4449b2c7fb4df86cbf041663971a91 Mon Sep 17 00:00:00 2001 From: Mark Phelps <209477+markphelps@users.noreply.github.com> Date: Wed, 25 Feb 2026 20:36:07 -0500 Subject: [PATCH 5/5] Apply suggestions from code review Co-authored-by: Roman Dmytrenko --- .../operations/deployment/kubernetes-troubleshooting.mdx | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/docs/v2/guides/operations/deployment/kubernetes-troubleshooting.mdx b/docs/v2/guides/operations/deployment/kubernetes-troubleshooting.mdx index 1bebe1e..3bb978f 100644 --- a/docs/v2/guides/operations/deployment/kubernetes-troubleshooting.mdx +++ b/docs/v2/guides/operations/deployment/kubernetes-troubleshooting.mdx @@ -13,8 +13,6 @@ When installing or upgrading the Flipt Helm chart, you may encounter schema vali helm install flipt flipt/flipt-v2 -f values.yaml Error: INSTALLATION FAILED: values don't meet the specifications of the schema(s) in the following chart(s): flipt-v2: -- (root): Additional property pgb is not allowed -- flipt.config.storage.bitbucket.backend: Additional property local is not allowed - autoscaling: Additional property targetMemoryUtilizationPercentage is not allowed - resources: Additional property limits is not allowed - resources: Additional property requests is not allowed @@ -129,8 +127,7 @@ flipt: my-storage: backend: type: local - local: - path: /var/opt/flipt + path: /var/opt/flipt ```