Skip to content

Comments

sensei: align validation with agentskills.io specification#996

Open
spboyer wants to merge 3 commits intomainfrom
sensei-spec-alignment
Open

sensei: align validation with agentskills.io specification#996
spboyer wants to merge 3 commits intomainfrom
sensei-spec-alignment

Conversation

@spboyer
Copy link
Member

@spboyer spboyer commented Feb 21, 2026 •
edited
Loading

Aligns sensei validation with the agentskills.io specification. See commit message for details.

  • Add missing name validation rules: no consecutive hyphens, no start/end hyphen, alphanumeric chars
  • Add Invalid score level for spec-failing names
  •  Strongly recommended (reported as suggestions if missing):   - license — identifies the license applied to the skill
  • metadata.version — tracks the skill version for consumers

@spboyer @Copilot
sensei: align validation with agentskills.io specification
de0c533 
- Add missing name validation rules: no consecutive hyphens, no start/end hyphen, alphanumeric chars
- Add Invalid score level for spec-failing names
- Reference agentskills.io/specification as authoritative source throughout
- Document optional spec fields (license, metadata, allowed-tools) and preserve-if-present policy
- Add SKILL.md 500-line limit check from spec
- Update scoring algorithm with isValidName function
- Update LOOP.md SCORE step with full spec checks
- Update README.md rule-based checks section

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings February 21, 2026 00:16
@spboyer @Copilot
sensei: strongly recommend license and metadata.version fields
fd75dbe 
- Elevate license and metadata.version from preserve-only to strongly recommended
- Report suggestions in summary when either field is missing
- Add collectSuggestions() to scoring algorithm
- Update frontmatter template to show license and version as defaults
- Add SPEC RECOMMENDATIONS section to summary display format
- Update High adherence example with license and metadata fields

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions
Copy link
Contributor

github-actions bot commented Feb 21, 2026 •
edited
Loading

🔍 Token Analysis Report

@github-copilot-for-azure/scripts@1.0.0 tokens
node --import tsx src/tokens/cli.ts compare --base origin/main --head HEAD --markdown

📊 Token Change Report

Comparing origin/main → HEAD

Summary

Metric Value
📈 Total Change +1,467 tokens (+14%)
Before 10,517 tokens
After 11,984 tokens
Files Changed 4

Changed Files

File Before After Change
.github/skills/sensei/references/SCORING.md 1,690 2,521 +831 (+49%)
.github/skills/sensei/references/LOOP.md 3,441 3,717 +276 (+8%)
.github/skills/sensei/SKILL.md 1,958 2,216 +258 (+13%)
.github/skills/sensei/README.md 3,428 3,530 +102 (+3%)

@github-copilot-for-azure/scripts@1.0.0 tokens
node --import tsx src/tokens/cli.ts check --markdown

📊 Token Limit Check Report

Checked: 391 files
Exceeded: 91 files

⚠️ Files Exceeding Token Limits

File Tokens Limit Over By
.github/skills/file-test-bug/SKILL.md 613 500 +113
.github/skills/sensei/README.md 3530 1000 +2530
.github/skills/sensei/SKILL.md 2216 500 +1716
.github/skills/sensei/references/EXAMPLES.md 3328 1000 +2328
.github/skills/sensei/references/LOOP.md 3717 1000 +2717
.github/skills/sensei/references/SCORING.md 2521 1000 +1521
.github/skills/sensei/references/TOKEN-INTEGRATION.md 1094 1000 +94
.github/skills/skill-authoring/SKILL.md 732 500 +232
plugin/skills/appinsights-instrumentation/SKILL.md 952 500 +452
plugin/skills/azure-ai/SKILL.md 835 500 +335
plugin/skills/azure-aigateway/SKILL.md 6329 500 +5829
plugin/skills/azure-compliance/SKILL.md 1238 500 +738
plugin/skills/azure-compliance/references/azqr-recommendations.md 1447 1000 +447
plugin/skills/azure-compliance/references/azqr-remediation-patterns.md 1987 1000 +987
plugin/skills/azure-compliance/references/azure-keyvault-expiration-audit.md 1286 1000 +286
plugin/skills/azure-compliance/references/azure-quick-review.md 1268 1000 +268
plugin/skills/azure-cost-optimization/SKILL.md 3456 500 +2956
plugin/skills/azure-deploy/SKILL.md 961 500 +461
plugin/skills/azure-deploy/references/pre-deploy-checklist.md 1004 1000 +4
plugin/skills/azure-deploy/references/troubleshooting.md 1123 1000 +123
plugin/skills/azure-diagnostics/SKILL.md 876 500 +376
plugin/skills/azure-kusto/SKILL.md 2167 500 +1667
plugin/skills/azure-messaging/SKILL.md 857 500 +357
plugin/skills/azure-messaging/references/service-troubleshooting.md 1044 1000 +44
plugin/skills/azure-observability/SKILL.md 939 500 +439
plugin/skills/azure-postgres/SKILL.md 1519 500 +1019
plugin/skills/azure-postgres/references/entra-rbac-overview.md 1823 1000 +823
plugin/skills/azure-postgres/references/group-sync.md 1844 1000 +844
plugin/skills/azure-postgres/references/permission-templates.md 1647 1000 +647
plugin/skills/azure-postgres/references/troubleshooting.md 1959 1000 +959
plugin/skills/azure-prepare/SKILL.md 1545 500 +1045
plugin/skills/azure-prepare/references/aspire.md 2573 1000 +1573
plugin/skills/azure-prepare/references/azure-context.md 1019 1000 +19
plugin/skills/azure-prepare/references/recipes/azd/aspire.md 1289 1000 +289
plugin/skills/azure-prepare/references/recipes/azd/azure-yaml.md 1709 1000 +709
plugin/skills/azure-prepare/references/recipes/azd/terraform.md 2924 1000 +1924
plugin/skills/azure-prepare/references/research.md 1600 1000 +600
plugin/skills/azure-prepare/references/runtimes/nodejs.md 1508 1000 +508
plugin/skills/azure-prepare/references/security.md 1784 1000 +784
plugin/skills/azure-prepare/references/services/functions/bicep.md 1896 1000 +896
plugin/skills/azure-prepare/references/services/functions/templates/SPEC-composable-templates.md 6187 1000 +5187
plugin/skills/azure-prepare/references/services/functions/templates/recipes/README.md 1265 1000 +265
plugin/skills/azure-prepare/references/services/functions/templates/recipes/common/uami-bindings.md 1223 1000 +223
plugin/skills/azure-prepare/references/services/functions/templates/recipes/composition.md 3150 1000 +2150
plugin/skills/azure-prepare/references/services/functions/templates/recipes/cosmosdb/README.md 1467 1000 +467
plugin/skills/azure-prepare/references/services/functions/templates/recipes/durable/README.md 1149 1000 +149
plugin/skills/azure-prepare/references/services/functions/templates/recipes/eventhubs/README.md 1403 1000 +403
plugin/skills/azure-prepare/references/services/functions/templates/recipes/mcp/source/java.md 1312 1000 +312
plugin/skills/azure-prepare/references/services/functions/templates/recipes/mcp/source/python.md 1207 1000 +207
plugin/skills/azure-prepare/references/services/functions/templates/recipes/mcp/source/typescript.md 1075 1000 +75
plugin/skills/azure-prepare/references/services/functions/templates/recipes/servicebus/README.md 1171 1000 +171
plugin/skills/azure-prepare/references/services/functions/templates/recipes/servicebus/source/dotnet.md 1227 1000 +227
plugin/skills/azure-prepare/references/services/functions/templates/recipes/sql/source/java.md 1009 1000 +9
plugin/skills/azure-prepare/references/services/functions/templates/recipes/sql/source/python.md 1080 1000 +80
plugin/skills/azure-prepare/references/services/functions/terraform.md 2305 1000 +1305
plugin/skills/azure-prepare/references/services/service-bus/patterns.md 1010 1000 +10
plugin/skills/azure-resource-lookup/SKILL.md 1379 500 +879
plugin/skills/azure-resource-lookup/references/azure-resource-graph.md 1307 1000 +307
plugin/skills/azure-resource-visualizer/SKILL.md 2091 500 +1591
plugin/skills/azure-storage/SKILL.md 1092 500 +592
plugin/skills/azure-storage/references/sdk-usage.md 1096 1000 +96
plugin/skills/azure-validate/SKILL.md 734 500 +234
plugin/skills/entra-app-registration/SKILL.md 2055 500 +1555
plugin/skills/entra-app-registration/references/api-permissions.md 2545 1000 +1545
plugin/skills/entra-app-registration/references/cli-commands.md 2211 1000 +1211
plugin/skills/entra-app-registration/references/console-app-example.md 2752 1000 +1752
plugin/skills/entra-app-registration/references/first-app-registration.md 1846 1000 +846
plugin/skills/entra-app-registration/references/oauth-flows.md 2375 1000 +1375
plugin/skills/entra-app-registration/references/troubleshooting.md 1896 1000 +896
plugin/skills/microsoft-foundry/SKILL.md 1912 500 +1412
plugin/skills/microsoft-foundry/foundry-agent/create/agent-framework/SKILL.md 1671 500 +1171
plugin/skills/microsoft-foundry/foundry-agent/create/agent-framework/references/debug-setup.md 1689 1000 +689
plugin/skills/microsoft-foundry/foundry-agent/deploy/deploy.md 3295 1000 +2295
plugin/skills/microsoft-foundry/foundry-agent/invoke/invoke.md 1273 1000 +273
plugin/skills/microsoft-foundry/foundry-agent/troubleshoot/troubleshoot.md 1299 1000 +299
plugin/skills/microsoft-foundry/models/deploy-model/SKILL.md 1627 500 +1127
plugin/skills/microsoft-foundry/models/deploy-model/capacity/SKILL.md 1725 500 +1225
plugin/skills/microsoft-foundry/models/deploy-model/customize/SKILL.md 2220 500 +1720
plugin/skills/microsoft-foundry/models/deploy-model/customize/references/customize-workflow.md 1859 1000 +859
plugin/skills/microsoft-foundry/models/deploy-model/preset/SKILL.md 1212 500 +712
plugin/skills/microsoft-foundry/models/deploy-model/preset/references/preset-workflow.md 4252 1000 +3252
plugin/skills/microsoft-foundry/models/deploy-model/preset/references/workflow.md 1315 1000 +315
plugin/skills/microsoft-foundry/project/create/create-foundry-project.md 1218 1000 +218
plugin/skills/microsoft-foundry/quota/references/ptu-guide.md 1473 1000 +473
plugin/skills/microsoft-foundry/quota/references/troubleshooting.md 1807 1000 +807
plugin/skills/microsoft-foundry/quota/references/workflows.md 1614 1000 +614
plugin/skills/microsoft-foundry/rbac/rbac.md 1752 1000 +752
plugin/skills/microsoft-foundry/references/sdk/foundry-sdk-py.md 1888 1000 +888
plugin/skills/microsoft-foundry/resource/create/create-foundry-resource.md 1489 1000 +489
plugin/skills/microsoft-foundry/resource/create/references/workflows.md 1637 1000 +637
.github/agents/SkillCreator.agent.md 1044 1000 +44

Consider moving content to references/ subdirectories.

Automated token analysis. See skill authoring guidelines for best practices.

Copilot AI reviewed Feb 21, 2026
View reviewed changes
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR aligns sensei skill validation with the agentskills.io specification. The changes introduce stricter name validation rules, document optional spec fields that must be preserved, add size limit checks, and introduce an "Invalid" scoring level for skills that fail spec validation.

Changes:

  • Added comprehensive name validation rules (no consecutive hyphens, no start/end hyphen, lowercase alphanumeric only) aligned with agentskills.io spec
  • Introduced "Invalid" scoring level for skills failing spec validation as a fail-fast mechanism
  • Documented optional spec fields (license, metadata, allowed-tools) that must be preserved during improvements
  • Added size limit documentation (500 line recommendation, 1024 char description hard limit, compatibility field max 500 chars)

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File Description
.github/skills/sensei/references/SCORING.md Added spec-aligned name validation rules, optional field preservation guidance, size limits table, and isValidName() function to scoring algorithm
.github/skills/sensei/references/LOOP.md Updated SCORE step with spec-based validation checks, added line count check to TOKENS step
.github/skills/sensei/SKILL.md Added "Invalid" score level, spec references throughout, documented optional fields in frontmatter template
.github/skills/sensei/README.md Updated rule-based checks section with spec-aligned validation rules and size limits
Comments suppressed due to low confidence (1)

.github/skills/sensei/references/SCORING.md:243

  • The regex pattern /^[a-z0-9][a-z0-9-]*[a-z0-9]$|^[a-z0-9]$/ does not properly enforce "no consecutive hyphens" constraint. A name like "azure--deploy" would match the first alternative since [a-z0-9-]* allows any number of hyphens including consecutive ones. The consecutive hyphen check on line 244 is redundant if the regex was correct, but necessary because the regex is insufficient. Consider either:
  1. Using a negative lookahead in the regex to prevent consecutive hyphens: /^[a-z0-9]([a-z0-9]|-(?!-))*[a-z0-9]$|^[a-z0-9]$/
  2. Or keep the current approach but add a comment explaining why the separate check on line 244 is necessary (the regex doesn't prevent consecutive hyphens)
        score = "Medium"

@spboyer @Copilot
sensei: add 1024-char description hard limit to scoring algorithm
427fe7c 
Addresses review feedback — the scoring algorithm now returns Invalid
if description exceeds the spec's 1024-character hard limit.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@spboyer
Copy link
Member Author

spboyer commented Feb 21, 2026

Review feedback addressed:

SCORING.md line 220: scoring algorithm missing 1024-char description hard limit check

Fixed in 427fe7c — added a check for the 1024-character hard limit before the 150-char minimum check. The algorithm now returns Invalid if the description exceeds 1024 characters, consistent with how name validation failures are handled.

spboyer added a commit to spboyer/sensei that referenced this pull request Feb 21, 2026
@spboyer @Copilot
fix: add 1024-char description hard limit to scoring algorithm
9deff9f 
The scoring algorithm pseudocode checked < 150 chars for Low but did not
guard against exceeding the agentskills.io spec hard limit of 1024 chars.
Added 'Invalid' level returned when description > 1024 characters.

Addresses feedback from microsoft/GitHub-Copilot-for-Azure#996.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@spboyer