Skip to content

Fix: Add dummy volume to force k8s-novolume hook prepare script execution #26

@rkoster

Description

@rkoster

Summary

Add a dummy EmptyDir volume mount to the cached-privileged-kubernetes container mode to work around a bug in GitHub's runner-container-hooks that prevents /github/workflow/event.json from being populated.

Problem

When using cached-privileged-kubernetes mode, the "Set up Docker Buildx" action fails because /github/workflow/event.json does not exist. This file is required by Docker Buildx to read the $GITHUB_EVENT_PATH environment variable.

Root Cause Analysis

The bug is in GitHub's runner-container-hooks package at packages/k8s/src/hooks/prepare-job.ts (lines 101-138):

let prepareScript: { containerPath: string; runnerPath: string } | undefined
if (args.container?.userMountVolumes?.length) {
    prepareScript = prepareJobScript(args.container.userMountVolumes || [])
}

// ... after copying workspace ...

if (prepareScript) {
    await execPodStep(
        ['sh', '-e', prepareScript.containerPath],
        createdPod.metadata.name,
        JOB_CONTAINER_NAME
    )
}

The prepare script (which copies /github/workflow and /github/home content) only gets created and executed if there are userMountVolumes.

From k8s/utils.ts (lines 35-55), the prepareJobScript function does:

#!/bin/sh -l
set -e
cp -R /__w/_temp/_github_home /github/home
cp -R /__w/_temp/_github_workflow /github/workflow
mkdir -p ${mountDirs}  # ← This comes from userMountVolumes

Why This Fails in Deskrun

  1. ✅ k8s-novolume hook creates /github emptyDir volume
  2. ✅ Hook copies /__w/ from runner to workflow pod
  3. ✅ Pod's /__w/_temp/_github_workflow/ contains event.json
  4. Prepare script is NEVER CREATED because no userMountVolumes exist
  5. ❌ Content never gets copied from /__w/_temp/_github_workflow/ to /github/workflow/
  6. ❌ Docker Buildx fails trying to access $GITHUB_EVENT_PATH (/github/workflow/event.json)

Debugging Evidence

Using kubectl exec on a workflow pod:

/github directory (EMPTY):

total 8
drwxrwsrwx    2 0        0             4096 Dec 29 09:43 .
drwxr-xr-x    1 0        0             4096 Dec 29 09:44 ..

/github/workflow directory:

Directory does not exist or is empty

/__w directory (has content):

total 28
drwxr-xr-x    7 1001     1001          4096 Dec 29 09:43 .
drwxr-xr-x    3 1001     1001          4096 Dec 29 09:43 _PipelineMapping
drwxr-xr-x    4 1001     1001          4096 Dec 29 09:43 _actions
drwxr-xr-x    8 1001     1001          4096 Dec 29 09:45 _temp
drwxr-xr-x    2 1001     1001          4096 Dec 29 09:42 _tool
drwxr-xr-x    3 1001     1001          4096 Dec 29 09:42 instant-cf

The /__w/_temp/_github_workflow/event.json exists but never gets copied to /github/workflow/.

Proposed Solution

Add a dummy EmptyDir volume mount to force the prepare script to execute. This is a minimal workaround that leverages existing hook behavior.

Changes to /pkg/templates/templates/overlay.yaml

In the build_hook_extension_spec() function:

1. Add dummy volume mount (after line 77):

#@ # WORKAROUND: Add dummy user mount volume to force k8s-novolume hook prepare script execution
#@ # The prepare script only runs when userMountVolumes exist, but it's needed to copy /github/workflow content
#@ # See: https://github.com/rkoster/deskrun/issues/XX
#@ volumeMounts.append({"name": "dummy-prepare-trigger", "mountPath": "/tmp/dummy-prepare"})

2. Add dummy volume (after line 110):

#@ # WORKAROUND: Dummy volume to trigger prepare script (see volumeMounts comment above)
#@ volumes.append({"name": "dummy-prepare-trigger", "emptyDir": {}})

Why This Works

  • The prepareJobScript function handles empty userMountVolumes gracefully
  • Adding any volume mount triggers the conditional and creates the prepare script
  • The prepare script copies GitHub workspace directories as a side effect
  • EmptyDir at /tmp/dummy-prepare is harmless and ephemeral

Scope

  • Affected Mode: cached-privileged-kubernetes only
  • Other Modes: dind and kubernetes are unaffected (different code paths)

Testing Plan

  1. Deploy updated template with dummy volume
  2. Verify workflow pod has both dummy volume and populated /github/workflow/
  3. Confirm /github/workflow/event.json exists with correct content
  4. Test Docker Buildx action succeeds
  5. Verify no regression in other container modes

Future Work

  • Report upstream bug to actions/runner-container-hooks repository
  • Remove workaround once upstream fix is available

Impact Assessment

Component Impact
cached-privileged-kubernetes mode Fixed
dind mode No change
kubernetes mode No change
Existing user configurations No change required

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions