mirror of
https://github.com/wshobson/agents
synced 2026-06-21 14:13:58 +00:00
Seth Hobson
cc37bfdd29
Add Claude Fable 5 (model: fable) as the tier above opus across the adapter framework - native passthrough in Claude Code, mapped to each harness's top model elsewhere. No agents are promoted yet; tier criteria and cost guidance are documented in docs/agents.md and docs/authoring.md. Refresh MODEL_ALIASES against current harness catalogs (June 2026): - Codex: gpt-5 family -> gpt-5.5 / gpt-5.4-mini (per Codex model docs) - Copilot: GPT-5 stand-ins -> native Claude models (claude-opus-4.8, claude-sonnet-4.6, claude-haiku-4.5) now that Copilot CLI serves them - OpenCode: opus -> anthropic/claude-opus-4-8; haiku dated pin -> latest-tracking anthropic/claude-haiku-4-5 - Gemini: intentionally stays on GA gemini-2.5-* (3.x is preview-gated) - plugin-eval judge opus tier -> claude-opus-4-8 Gates: generate-all (no committed drift), validate STRICT=1, garden, test (442 passed), smoke-test all green.
plugin-eval
Three-layer quality evaluation framework for Claude Code plugins.
Quick Start
cd plugins/plugin-eval
uv sync
# Evaluate a skill (static only, instant)
uv run plugin-eval score path/to/skill --depth quick
# Evaluate with LLM judge (~30s)
uv run plugin-eval score path/to/skill --depth standard
# Full certification (all layers, ~5 min)
uv run plugin-eval certify path/to/skill
Layers
- Static Analysis — Structural checks, anti-pattern detection. Instant, free.
- LLM Judge — Semantic evaluation (triggering, orchestration, output, scope). ~30s, 4 calls.
- Monte Carlo — Statistical reliability via 50–100 simulated runs. ~2–5 min.
Commands
| CLI | Claude Code | Description |
|---|---|---|
plugin-eval score |
/eval |
Score a plugin or skill |
plugin-eval certify |
/certify |
Full certification with badge |
plugin-eval compare |
/compare |
Head-to-head comparison |
plugin-eval init |
— | Build corpus for Elo ranking |
Documentation
See docs/plugin-eval.md for the full reference: layers, dimensions, scoring formula, anti-patterns, statistical methods, and project structure.