Add azure-hosted-copilot-sdk skill with test infrastructure and local dev tooling

[jongio]

Adds the azure-hosted-copilot-sdk skill that guides users through building apps with the GitHub Copilot SDK (@github/copilot-sdk) and deploying them to Azure. Uses a single template (azure-samples/copilot-sdk-service) with three model paths: GitHub default, GitHub specific, and Azure BYOM (Bring Your Own Model).

Feature spec: docs/spec/azure-hosted-copilot-sdk.md

What's Changed

New Skill: azure-hosted-copilot-sdk

• SKILL.md — Routes users through scaffold → deploy → model config workflows
• references/copilot-sdk.md — SDK reference, template structure, testing guidance
• references/azure-model-config.md — Three model paths: GitHub default, GitHub specific, Azure BYOM with DefaultAzureCredential
• references/deploy-existing.md — Deploy workflow, Bicep infrastructure, token flow, Dockerfile patterns
• references/existing-project-integration.md — Add Copilot SDK to existing apps, BYOM integration

Tests: azure-hosted-copilot-sdk

• triggers.test.ts — 23 tests (10 trigger, 8 negative, 5 edge cases)
• unit.test.ts — 17 tests (metadata, content validation, BYOM patterns, frontmatter)
• integration.test.ts — 4 invocation rate + 2 content quality tests

Test Infrastructure (tests/utils/)

• agent-runner.ts — Enhanced with runConversation() for multi-turn test scenarios, per-turn and aggregate metadata tracking
• regression-detectors.ts — Reusable detector functions for common deployment anti-patterns: secret leaks, ACR auth spirals, port-binding confusion, hosting thrashing
• evaluate.ts — Shared evaluation helpers extracted from test files
• git-clone.ts — Git clone utility for test fixtures

Test Templates (tests/_template/)

• Updated integration.test.ts template with improved patterns
• Added unit.test.ts template for consistent test structure across skills

Local Dev Tooling (scripts/src/local/)

• test command — Live validation of the local plugin against real agent sessions
• setup command — Enhanced with MCP server registration (mcp-config.json)
• verify command — Expanded health checks for plugin and MCP servers

Reference Validation (scripts/src/references/)

• cli.ts — Validates markdown references, detects broken links, orphaned refs, and out-of-place files
• references.test.ts — Tests for reference validation logic

Other Changes

• plugin/.mcp.json — Added MCP server entries for Microsoft Learn
• .gitignore — Updated with test output patterns
• CI workflows — Added test-all-integration.yml, test-all-skills.yml, updated pr.yml
• Existing skill updates — Various skills updated with SDK references, resource graph integration, and region availability docs (carried from upstream rebases)

Test Plan

• Unit and trigger tests: npm test -- tests/azure-hosted-copilot-sdk/
• All skill tests: npm test -- tests/
• Local plugin validation: npm run local test

[Copilot]

Pull request overview

Adds a new github-copilot skill (Copilot SDK + Copilot Extensions, Azure deployment pathing) and expands the test and local-dev infrastructure to support multi-turn deployment scenarios, plus documentation improvements for validate/deploy workflows.

Changes:

• Introduces plugin/skills/github-copilot with references for Copilot SDK agent and Copilot Extensions.
• Adds new integration tests for multi-turn GHCP SDK → Azure deployment scenarios, including regression detectors.
• Enhances local dev scripts to register MCP servers from plugin/.mcp.json and adds a live “local test” command; updates azure-validate/azure-deploy docs/checklists.

Reviewed changes

Copilot reviewed 31 out of 31 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
tests/utils/regression-detectors.ts	Adds event-scanning detectors for common failure/regression patterns in deployment conversations.
tests/utils/ghcp-sdk-workspace.ts	Adds helper to scaffold an azd Copilot SDK agent template into a test workspace.
tests/utils/agent-runner.ts	Adds runConversation() multi-turn runner and minor filename sanitization fix.
tests/github-copilot/unit.test.ts	Adds unit tests validating the new skill’s metadata/content/frontmatter.
tests/github-copilot/triggers.test.ts	Adds trigger/anti-trigger tests and snapshots for the new skill.
tests/github-copilot/integration.test.ts	Adds integration tests intended to validate real session behavior for the new skill.
tests/github-copilot/fixtures/sample.json	Adds sample prompts/expectations and regression limits for scenarios.
tests/github-copilot/snapshots/triggers.test.ts.snap	Adds snapshots for extracted trigger keywords/description.
tests/github-copilot/README.md	Adds a README (currently template-style) under the new skill test folder.
tests/ghcp-sdk-foundry-agent/integration.test.ts	Adds multi-turn end-to-end integration test for Foundry-hosted agent scenario.
tests/ghcp-sdk-deploy/integration.test.ts	Adds multi-turn end-to-end integration test for GHCP SDK Azure deployment scenario.
tests/_template/unit.test.ts	Updates the test template with frontmatter-format validation checks.
tests/README.md	Updates the skill coverage grid with additional skill rows.
scripts/src/local/commands/verify.ts	Adds MCP server registration checks and validates skill SKILL.md frontmatter existence.
scripts/src/local/commands/test.ts	Adds a live smoke-test command that runs Copilot CLI prompts to validate plugin + MCP connectivity.
scripts/src/local/commands/setup.ts	Extends setup to merge plugin/.mcp.json into ~/.copilot/mcp-config.json.
scripts/src/local/cli.ts	Adds npm run local test command wiring.
plugin/skills/github-copilot/references/copilot-sdk-agent.md	Adds Copilot SDK agent reference links and workflow guidance.
plugin/skills/github-copilot/references/copilot-extensions.md	Adds Copilot Extensions reference links and operational guidance (SSE/signature verification).
plugin/skills/github-copilot/SKILL.md	Adds the new github-copilot skill with path detection + deployment orchestration rules.
plugin/skills/azure-validate/references/recipes/azd/README.md	Inserts explicit “Build Verification” step before packaging validation.
plugin/skills/azure-validate/SKILL.md	Updates validate steps to include build verification before proceeding.
plugin/skills/azure-deploy/references/recipes/azd/errors.md	Adds troubleshooting guidance for duplicate azd-service-name tagged resources.
plugin/skills/azure-deploy/references/recipes/azd/README.md	Adds guidance to stream azd up output visibly to the user.
plugin/skills/azure-deploy/references/pre-deploy-checklist.md	Updates duplicate-tag resolution guidance (prefer new env, else delete with confirmation).
plugin/skills/azure-deploy/SKILL.md	Adds guidance to stream deployment output visibly.
plugin/.mcp.json	Adds additional MCP servers (context7, playwright, microsoft-learn).
.github/skills/sensei/references/SCORING.md	Updates frontmatter format guidance, especially around USE FOR/DO NOT USE FOR parsing.
.github/skills/sensei/references/LOOP.md	Adds detection/fix guidance for unsafe frontmatter formatting cases.
.github/skills/sensei/references/EXAMPLES.md	Updates examples to prefer quoted descriptions for USE FOR/DO NOT USE FOR content.
.github/skills/sensei/SKILL.md	Updates the Sensei loop/template guidance accordingly.

[kvenkatrajan]

Please address copilot comments and checks are failing. Had one comment on the yaml description. Please add screenshot of integration test result

[tmeschter]

Approved with some comments.

[JasonYeMSFT]

Please fix the type errors reported by the PR Checks.

[spboyer]

Feedback on Sensei Pipe Block Scalar Changes

The sensei changes in this PR convert all description: | (pipe block scalar) examples to quoted description: ""..."" format, citing that the Copilot CLI parser misinterprets USE FOR: and DO NOT USE FOR: on indented continuation lines as top-level YAML keys.

This conflicts with existing guidance and specs

1. Our own guidance doc (Skills, Tools and MCP Development Guide) explicitly recommends description: | with USE FOR: / DO NOT USE FOR: / INVOKES: on indented lines as the canonical pattern.

2. The agentskills.io specification allows any valid YAML scalar format for the description field -- plain, quoted, or block scalar. It does not restrict pipe usage.

3. Standard YAML behavior -- in valid YAML, when using description: |, all indented lines are part of the string value. USE FOR: on an indented line is never a top-level key. Any parser that interprets it that way has a bug.

What the repo looks like today

We currently have a mix of formats across skills:

• Pipe (10 skills): microsoft-foundry, azure-role-selector, azure-resource-visualizer, entra-app-registration, azure-aigateway, azure-diagnostics, azure-networking, azure-kusto, azure-compliance, azure-cost-optimization
• Quoted (6 skills): azure-validate, azure-prepare, azure-deploy, azure-ai, appinsights-instrumentation
• Plain unquoted (2 skills): azure-storage, azure-postgres, azure-observability

The pipe-format skills are currently working, which means either the parser bug is intermittent or has been fixed.

Recommendation

The fix here is a workaround, not the right solution. A 600+ character description on a single quoted line is hard to read, review, and maintain -- which is exactly why we recommended the pipe format in our guidance doc.

The right approach:

1. File a bug against the Copilot CLI parser if it is genuinely misinterpreting indented continuation lines as top-level YAML keys (I attempted to file at github/copilot-cli but need SSO auth for that org)
2. Keep description: | as the recommended format for multi-line descriptions with USE FOR: / DO NOT USE FOR:
3. If the parser bug is confirmed and will not be fixed soon, acknowledge the workaround as a temporary parser limitation in the sensei docs, not as a YAML best practice

Happy to discuss -- just want to make sure we are not encoding a parser bug as permanent guidance.

cc @jongio

[spboyer]

Update: Filed the parser bug as github/copilot-cli#1427 — the YAML frontmatter parser misinterprets pipe block scalar continuation lines containing USE FOR: / DO NOT USE FOR: as top-level keys. This should be fixed in the parser rather than worked around in skill authoring guidance.

[Copilot]

Pull request overview

Copilot reviewed 41 out of 42 changed files in this pull request and generated 6 comments.

Comments suppressed due to low confidence (1)

tests/README.md:438

• The Skills Coverage Grid in tests/README.md is missing an entry for the new 'github-copilot' skill. Since this PR adds comprehensive test coverage for this skill (unit.test.ts, triggers.test.ts, integration.test.ts), it should be added to the grid with appropriate checkmarks. Add a row like: | github-copilot | ✅ | ✅ | ✅ | ✅ | - |

| Skill | Tests | Unit | Triggers | Integration | Coverage |
|-------|-------|------|----------|-------------|----------|
| appinsights-instrumentation | ✅ | ✅ | ✅ | ✅ | - |
| azure-ai | ✅ | ✅ | ✅ | ✅ | - |
| azure-aigateway | ✅ | ✅ | ✅ | ✅ | - |
| azure-cost-optimization | ✅ | ✅ | ✅ | ✅ | - |
| azure-create-app | ❌ | - | - | - | - |
| azure-deploy | ✅ | ✅ | ✅ | ✅ | - |
| azure-deployment-preflight | ❌ | - | - | - | - |
| azure-diagnostics | ✅ | - | - | ✅ | - |
| azure-functions | ✅ | - | - | ✅ | - |
| azure-keyvault-expiration-audit | ✅ | ✅ | ✅ | ✅ | - |
| azure-kusto | ✅ | - | - | ✅ | - |
| azure-nodejs-production | ✅ | - | - | ✅ | - |
| azure-observability | ✅ | - | - | ✅ | - |
| azure-postgres | ✅ | - | - | ✅ | - |
| azure-prepare | ✅ | - | - | ✅ | - |
| azure-quick-review | ✅ | ✅ | ✅ | ✅ | - |
| azure-resource-visualizer | ✅ | - | - | ✅ | - |
| azure-role-selector | ✅ | ✅ | ✅ | ✅ | - |
| azure-security | ✅ | - | - | ✅ | - |
| azure-security-hardening | ✅ | - | - | ✅ | - |
| azure-storage | ✅ | - | - | ✅ | - |
| azure-validate | ✅ | ✅ | ✅ | ✅ | - |
| entra-app-registration | ✅ | ✅ | ✅ | ✅ | - |
| microsoft-foundry | ✅ | ✅ | ✅ | ✅ | - |

[Copilot]

Pull request overview

Copilot reviewed 40 out of 41 changed files in this pull request and generated 6 comments.

[Copilot]

Pull request overview

Copilot reviewed 34 out of 35 changed files in this pull request and generated 2 comments.

[Copilot]

There's a typo in the error message. "doesn"t" should be "doesn't" (with proper apostrophe). This will appear in validation error output when a skill name doesn't match its directory name.

Suggested change

	invalid.push({ name: skill, error: `name "${nameMatch[1].trim()}" doesn"t match directory` });
	invalid.push({ name: skill, error: `name "${nameMatch[1].trim()}" doesn't match directory` });

[Copilot]

The "context7" and "playwright" MCP server entries are missing the required "type" field. Looking at the existing entries, "azure" has command-based configuration (stdio type), while "foundry-mcp" explicitly sets "type": "http". For consistency and proper configuration, stdio-based servers should include "type": "stdio".

Add the type field:

"context7": {
  "type": "stdio",
  "command": "npx",
  "args": ["-y", "@upstash/context7-mcp@latest"]
},
"playwright": {
  "type": "stdio",
  "command": "npx",
  "args": ["-y", "@playwright/mcp@latest"]
}

Suggested change

	"context7": {
	"command": "npx",
	"args": ["-y", "@upstash/context7-mcp@latest"]
	},
	"playwright": {
	"context7": {
	"type": "stdio",
	"command": "npx",
	"args": ["-y", "@upstash/context7-mcp@latest"]
	},
	"playwright": {
	"type": "stdio",

[Copilot]

Pull request overview

Copilot reviewed 39 out of 41 changed files in this pull request and generated 2 comments.

Files not reviewed (1)

• scripts/package-lock.json: Language not supported

[Copilot]

The test expects the routing table to say "Use Azure/own model → Step 2C" but the actual SKILL.md correctly routes it to "Step 3 (BYOM config)". The test assertion on line 95 should expect "Step 3" instead of "Step 2C" to match the actual routing logic in the skill.

[Copilot]

The package-lock.json file has added "peer": true flags to several dependencies (@types/node, @typescript-eslint/ast-spec, acorn, eslint-plugin-deprecation, fast-glob, typescript, typescript-eslint, vitest). The "peer" field in package-lock.json is used to indicate peer dependency metadata. These additions appear correct for properly tracking peer dependencies, though it's worth verifying that these are intentional changes and not accidental modifications from a dependency resolution change.

Suggested change

"peer": true,