Card Analyses: 6 Models for Publication

[PipFoweraker]

Purpose

Analyze 6 real model cards against interrogatory requirements to:

1. Demonstrate the framework works across model types
2. Show documentation gaps that the framework exposes
3. Provide concrete examples for LW/AF publication

Selected Models

Frontier (2)

• Claude (3.5 Sonnet or 4) - Anthropic system card - expected to score well
• GPT-4 (or 4o) - OpenAI system card - benchmark comparison

Research (2)

• Mistral 7B or Mixtral - European lab, reportedly sparse documentation
• Qwen 2.5 - Alibaba, Chinese lab norms, interesting comparison

Unusual (2)

• Whisper - Audio transcription, non-generative
• Llama Guard 3 - Safety classifier, meta-level model

Analysis Template

For each model:

1. Locate official card/documentation
2. Score against 14 MUST questions
3. Note SHOULD/CAN elements present
4. Calculate disclosure completeness %
5. Document specific gaps with quotes
6. Write brief narrative assessment

Output

• data/card-analysis/{model}-analysis.md for each model
• Summary comparison table
• Narrative for LW/AF post

Timeline

Target: Complete by Feb 7 to allow time for writeup before mid-Feb publication

[PipFoweraker]

Completed ✅

All 6 model analyses have been completed:

Analysis Files Created

• claude-3-5-sonnet-analysis.md - 14% full / 57% with partial
• gpt-4-analysis.md - 21% full / 64% with partial
• mistral-7b-analysis.md - 15% full / 15% with partial
• qwen-2.5-7b-analysis.md - 14% full / 29% with partial
• whisper-large-v3-analysis.md - 43% full / 93% with partial
• llama-guard-3-8b-analysis.md - 57% full / 100% with partial
• SYNTHESIS.md - Cross-model comparison and key findings
• TEMPLATE.md - Standardized analysis template

Key Findings

1. Average disclosure: 27% full, 60% with partial credit
2. Best performer: Llama Guard 3 at 57% (purpose-built safety model)
3. Worst performer: Mistral 7B at 15% (sparse documentation despite "open" positioning)
4. Universal gap: 0/6 models provide full reproducibility triad
5. Architecture gap: Only open-weight models disclose architecture; frontier models are silent

Notable Patterns

• Purpose-built models document better than general-purpose
• US labs document safety; non-US labs often don't
• Training data remains universally opaque
• Update policies are missing in 5/6 models

Ready for integration into LW/AF publication.