Add nabledge-6-test skill and execute first test scenario

[kiyotis]

Summary

This PR introduces a unified test framework nabledge-test for nabledge-6 and nabledge-5 skills, powered by skill-creator's evaluation engine.

Architecture

nabledge-test (Interface Layer)
  ├── Nablarch-specific scenarios
  ├── Scenario → Eval conversion
  └── Report generation
          ↓ delegates to
skill-creator (Evaluation Engine)
  ├── Executor agent (run skill)
  ├── Grader agent (evaluate expectations)
  └── Analyzer agent (pattern analysis)

Key Changes

1. Added skill-creator (Claude.ai Environment Version)

• Source: Claude.ai production environment (not GitHub anthropics/skills)
• Features:
  • Eval Mode: Execute individual evals with executor/grader agents
  • Benchmark Mode: Statistical analysis with 3x runs per configuration
  • Complete evaluation framework (transcript, metrics, grading)
  • Structured outputs for reproducible testing

Why Claude.ai version?

• GitHub version lacks eval/benchmark modes
• Claude.ai version has production-tested evaluation agents
• Full feature set: executor, grader, comparator, analyzer agents

2. Refactored nabledge-6-test → nabledge-test

Before: nabledge-6-test (monolithic)
After: nabledge-test (interface) + skill-creator (engine)

New command syntax:

# nabledge-6
nabledge-test 6 handlers-001
nabledge-test 6 --all
nabledge-test 6 --category handlers

# nabledge-5 (future)
nabledge-test 5 libraries-001

3. Architecture Benefits

Aspect	Before	After
Evaluation	Manual workflow	skill-creator agents
Scope	nabledge-6 only	nabledge-6 + nabledge-5
Engine	Custom implementation	Anthropic's proven framework
Maintenance	Reinventing wheel	Leverage existing tools
Features	Basic testing	Advanced analysis (variance, comparison)

4. New Components

nabledge-test/:

• SKILL.md - Nablarch-specific test interface
• scenarios/nabledge-6/scenarios.json - 30 test scenarios
• scenarios/nabledge-5/ - Future support
• scripts/convert_scenario.py - Convert to eval format

skill-creator/ (762 lines):

• agents/executor.md - Execute eval with skill
• agents/grader.md - Evaluate expectations
• agents/analyzer.md - Pattern analysis
• agents/comparator.md - A/B comparison
• references/eval-mode.md - Eval workflow
• references/benchmark-mode.md - Benchmark workflow
• scripts/*.py - Automation scripts

5. Workflow

1. Load scenario (scenarios/nabledge-6/scenarios.json)
   ↓
2. Convert to eval format
   {
     "prompt": "データリードハンドラでファイルを読み込むには？",
     "expectations": [
       "Response includes keyword 'DataReadHandler'",
       "Token usage is between 5000 and 15000"
     ]
   }
   ↓
3. Execute via skill-creator
   - Executor agent: Run nabledge-6 with prompt
   - Grader agent: Evaluate expectations
   ↓
4. Generate nabledge report
   work/YYYYMMDD/test-handlers-001-timestamp.md

Test Results (Pre-Integration)

Using original nabledge-6-test (before refactoring), handlers-001 scenario:

Criterion	Result	Details
Workflow Execution	✅ PASS	keyword-search, section-judgement executed
Keyword Matching	✅ PASS	80% (4/5)
Section Relevance	✅ PASS	Expected sections identified
Knowledge File Only	✅ PASS	Proper citations
Token Efficiency	❌ FAIL	20k tokens (target: 15k)
Tool Call Efficiency	✅ PASS	9 calls (target: 10-20)

Overall: 5/6 criteria passed (83%)

Migration Impact

For Users

Old command:

/nabledge-6-test execute handlers-001

New command:

/nabledge-test 6 handlers-001

For Developers

• 30 scenarios preserved in scenarios/nabledge-6/
• Report format unchanged (compatibility)
• Can now add nabledge-5 scenarios in scenarios/nabledge-5/

Why This Approach?

❌ Initial Approach (nabledge-6-test)

• Reimplemented test execution from scratch
• Custom evaluation logic
• Monolithic design
• Single version support

✅ New Approach (nabledge-test + skill-creator)

• Leverage Anthropic's evaluation framework
• Proven executor/grader/analyzer agents
• Layered architecture (interface vs engine)
• Multi-version support (6 & 5)
• Future-proof (skill-creator updates benefit us)

Future Work

1. nabledge-5 support: Add scenarios/nabledge-5/scenarios.json
2. Benchmark mode: Use skill-creator's statistical analysis (3x runs, variance)
3. A/B testing: Compare skill versions using comparator agent
4. Continuous improvement: Leverage analyzer agent patterns

Files Changed

• .claude/settings.json: Update skill permissions
• .claude/skills/nabledge-test/: New unified test framework
• .claude/skills/skill-creator/: Anthropic's evaluation engine (Claude.ai version)
• Removed: .claude/skills/nabledge-6-test/

🤖 Generated with Claude Code