feat: add specVersion classification to conformance scenarios#147
Merged
feat: add specVersion classification to conformance scenarios#147
Conversation
commit: |
felixweinberger
force-pushed
the
fweinberger/spec-version-classification
branch
2 times, most recently
from
4da4fb2 to
974ca0e
Compare
felixweinberger
changed the base branch from
main
to
fweinberger/fix-parseoutputdir-recursion
felixweinberger
force-pushed
the
fweinberger/spec-version-classification
branch
3 times, most recently
from
0af9522 to
a2bdeac
Compare
Base automatically changed from
fweinberger/fix-parseoutputdir-recursion
to
main
February 13, 2026 16:13
felixweinberger
force-pushed
the
fweinberger/spec-version-classification
branch
from
a2bdeac to
7ce9f91
Compare
Contributor
Author
|
Output previews:
|
felixweinberger
force-pushed
the
fweinberger/spec-version-classification
branch
from
7ce9f91 to
e5f080d
Compare
felixweinberger
force-pushed
the
fweinberger/spec-version-classification
branch
2 times, most recently
from
0b7797c to
879dffd
Compare
Each scenario declares which spec versions it applies to as a list. Scenarios that carry forward (e.g. initialize) list all applicable versions ['2025-06-18', '2025-11-25']. Scenarios removed from newer specs (e.g. backcompat auth) only list their original version ['2025-03-26']. - specVersions list on Scenario and ClientScenario interfaces - --spec-version CLI filter uses simple .includes() - Tier-check conformance matrix (Server / Client: Core / Client: Auth) with per-version columns and unique All* count - 7 unit tests for specVersions helpers - Updated tier-audit skill docs with matrix format
The template had an orphan table header (Check | Value | T2 | T1) with no rows above the conformance matrix, causing an empty table to render. The scorecard rows below the matrix also lacked their own header. Fix: two self-contained tables with clear labels — 'Conformance:' for the per-version matrix, 'Scorecard:' for the check rows.
Add asterisk footnote ('unique scenarios — a scenario may apply to
multiple spec versions') and rename 'Scorecard' to 'Repository Health'
to match the labels used by the tier-check CLI output.
Added in 83c446d on main after this branch diverged.
felixweinberger
force-pushed
the
fweinberger/spec-version-classification
branch
from
a1b19a3 to
dfa9036
Compare
pcarleton
approved these changes
Feb 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add a
specVersionfield toScenarioandClientScenariointerfaces so SDK maintainers and tier-check scoring can filter tests by target spec version.SpecVersiontype:2025-03-26|2025-06-18|2025-11-25|draft|extension--spec-versionCLI filter onclient,server, andlistcommands (cumulative for date versions, exact-match for draft/extension)listoutput shows[specVersion]tag next to each scenariolistScenariosForSpec,listScenariosUpToSpec,getScenarioSpecVersion, etc.Preview
Motivation and Context
PR #1777 (SEP-1730) defines an SDK tiering system where Tier 2 SDKs must pass 80% of tests for the 2025-06-18 spec and Tier 1 must pass 100% for 2025-11-25. Without metadata on scenarios indicating which spec version they target, SDK maintainers cannot know which tests apply to them, and the tier-check scoring logic cannot filter by target version.
How Has This Been Tested
pnpm build— clean build, no type errorspnpm test— 77/77 tests passnode dist/index.js list— all scenarios show spec version tags--spec-version 2025-06-18— cumulative filter works correctly--spec-version draft/extension— exact-match filter works correctlyspecVersionis required on all scenariosBreaking Changes
ScenarioandClientScenariointerfaces now require aspecVersionfield — any external implementations of these interfaces will need to add the fieldTypes of changes
Checklist