Add nightly agent CLI integration tests

[jwiegley]

Summary

• Adds .github/workflows/nightly-agent-integration.yml — a two-tier nightly workflow that installs real agent CLI binaries and verifies git-ai hook wiring and attribution end-to-end
• Adds scripts/nightly/ with four helper scripts implementing the test logic
• Adds NIGHTLY_INTEGRATION_PLAN.md documenting the full design rationale and open questions

Test Architecture

Tier 1 — Hook Wiring (no API keys, free)

Builds git-ai from source, installs each agent CLI (Claude Code, Codex, Gemini, Droid, OpenCode) at both stable and latest versions via a dynamic matrix, then:

1. Runs git-ai install and verifies the correct checkpoint commands appear in each agent's config file
2. Exercises the full attribution pipeline with synthetic checkpoint data (via the agent-v1 preset)

Tier 2 — Live Integration (requires API key secrets)

Runs each agent with a minimal deterministic prompt ("create hello.txt, commit it"), then verifies the file was created, a commit landed, and authorship notes are present in refs/notes/ai. Pre-release failures are non-blocking (continue-on-error: true).

Hook config paths (verified against src/mdm/agents/*.rs)

Agent	Config file
Claude Code	~/.claude/settings.json
Codex	~/.codex/config.toml
Gemini CLI	~/.gemini/settings.json
Droid	~/.factory/settings.json
OpenCode	~/.config/opencode/plugin/git-ai.ts

Secrets required (Tier 2 only)

ANTHROPIC_API_KEY, OPENAI_API_KEY, GEMINI_API_KEY, FACTORY_API_KEY, SLACK_BOT_TOKEN, SLACK_CHANNEL_ID

Tier 1 runs without any secrets.

Cost estimate

~$0.05–0.25/night (weekdays only). See NIGHTLY_INTEGRATION_PLAN.md §6 for cost management strategies.

Test plan

• Verify workflow YAML parses correctly in Actions UI
• Trigger workflow_dispatch with tier: tier1 to validate hook-wiring jobs (no API keys needed)
• Add ANTHROPIC_API_KEY secret and trigger tier: both to validate Claude Code Tier 2 end-to-end
• Review open questions in NIGHTLY_INTEGRATION_PLAN.md §13 before enabling the nightly schedule

🤖 Generated with Claude Code

---

[git-ai-cloud-dev]

Stats powered by Git AI

🧠 you    ████░░░░░░░░░░░░░░░░  22%
🤖 ai     ░░░░████████████████  78%

More stats

• 0.0 lines generated for every 1 accepted
• 0 seconds waiting for AI
• Top model: claude::claude-sonnet-4-6 (263 accepted lines, 0 generated lines)

AI code tracked with git-ai

[git-ai-cloud]

Stats powered by Git AI

🧠 you    ████░░░░░░░░░░░░░░░░  22%
🤖 ai     ░░░░████████████████  78%

More stats

• 0.9 lines generated for every 1 accepted
• 4 minutes waiting for AI
• Top model: claude::claude-sonnet-4-6 (263 accepted lines, 238 generated lines)

AI code tracked with git-ai

[devin-ai-integration]

Devin Review found 1 new potential issue.

View 28 additional findings in Devin Review.

[devin-ai-integration]

🟡 agents workflow_dispatch input is defined but never consumed by the matrix builder

The agents input allows users to specify "claude" or "claude,codex" when manually triggering the workflow, with the documented intent of filtering which agents to test. However, the Python matrix-builder script at lines 44–116 never reads github.event.inputs.agents — it unconditionally builds entries for all four npm agents plus Droid.

Root cause and impact

A user who triggers workflow_dispatch with agents: "claude" expecting to test only Claude will instead run the full matrix (all agents × all channels), wasting CI time and potentially burning API credits in Tier 2. The input parameter at nightly-agent-integration.yml:10-12 has no effect on the matrix output at nightly-agent-integration.yml:44-116.

Actual behavior: All agents are always included in the matrix regardless of the agents input value.

Expected behavior: When agents is not "all", only the specified agents should appear in the matrix.

Prompt for agents

In .github/workflows/nightly-agent-integration.yml, the Python matrix-builder script (lines 44-116) needs to read the agents workflow_dispatch input and filter accordingly. At the top of the Python script (around line 45), read the input:

  requested = os.environ.get("INPUT_AGENTS", "all").strip()
  requested_set = None if requested == "all" else set(a.strip() for a in requested.split(","))

Then, inside the for-loop over agents (around line 62), skip agents not in the requested set:

  if requested_set is not None and agent not in requested_set:
      continue

Similarly, conditionally include the Droid entry (around line 102) only when requested_set is None or "droid" in requested_set.

You also need to pass the input as an env var to the step. Add to the step at line 41:
  env:
    INPUT_AGENTS: ${{ github.event.inputs.agents || 'all' }}

---

Was this helpful? React with 👍 or 👎 to provide feedback.