Skip to content

Add AI-ready codebase checks (coverage + type enforcement) #32

@oldsj

Description

@oldsj

Background

Based on the article "AI Is Forcing Us To Write Good Code", AI agents require disciplined software practices:

  1. 100% Code Coverage - Forces every line to have executable examples
  2. Intentional File Organization - Semantic paths, small focused files
  3. Fast, Ephemeral, Concurrent Environments - Speed + isolation
  4. End-to-End Type Systems - TypeScript strict, OpenAPI, Pydantic

Gap Analysis: Mainloop vs Recommendations

Recommendation Current State Gap
Code Coverage ~6% test-to-code ratio, no reporting LARGE
Semantic File Organization Strong - no generic utils/ Minimal
TypeScript Strict Mode Enabled None
Python Type Hints Comprehensive with Pydantic None
Static Type Checking in CI Not enforced (no mypy/pyright) MODERATE
Strict Linting Excellent (Trunk) Minimal

Proposal: ai-ready-check GitHub Action

Create a reusable GitHub Action that enforces AI-readiness guardrails:

Checks to Implement

  1. Coverage Gate - Run pytest-cov, fail if coverage decreases (ratchet strategy)
  2. Type Check Gate - Run pyright (Python) + svelte-check (frontend)

Action Structure

.github/actions/ai-ready-check/
├── action.yml           # Action metadata and inputs
├── coverage/
│   ├── python.sh        # pytest-cov runner
│   └── compare.py       # Ratchet comparison logic
├── types/
│   ├── python.sh        # pyright runner
│   └── typescript.sh    # svelte-check runner
└── report.py            # Generate GitHub summary

action.yml Inputs

  • python-path: Path to Python source (default: backend/src)
  • typescript-path: Path to TypeScript source (default: frontend/src)
  • coverage-baseline-file: Path to baseline JSON (default: .coverage-baseline.json)
  • fail-on-coverage-decrease: Whether to fail if coverage drops (default: true)

Implementation Plan

Phase 1: Create Reusable Action Structure

  • Create action.yml with inputs
  • Create shell scripts for running coverage and type checks

Phase 2: Implement Coverage Check

  • Add pytest-cov to backend dev dependencies
  • Create ratchet comparison logic (fail only if coverage decreases)
  • Store baseline in .coverage-baseline.json

Phase 3: Implement Type Checks

  • Add pyright to backend dev dependencies
  • Run pnpm run check (svelte-check already exists in frontend)

Phase 4: Integrate with CI

  • Add ai-ready-check job to .github/workflows/ci.yml
  • Create initial coverage baseline

Files to Create/Modify

File Action
.github/actions/ai-ready-check/action.yml Create
.github/actions/ai-ready-check/coverage/python.sh Create
.github/actions/ai-ready-check/coverage/compare.py Create
.github/actions/ai-ready-check/types/python.sh Create
.github/actions/ai-ready-check/types/typescript.sh Create
.github/actions/ai-ready-check/report.py Create
backend/pyproject.toml Modify - add pytest-cov, pyright
.github/workflows/ci.yml Modify - add ai-ready-check job
.coverage-baseline.json Create - initial baseline

Success Criteria

  • Action runs on every PR
  • Coverage percentage shown in PR summary
  • Build fails if coverage decreases
  • Type errors block merge
  • Action is reusable by other repos

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions