mirror of
https://github.com/wshobson/agents
synced 2026-06-21 14:13:58 +00:00
Seth Hobson
0041fc7194
Adds a GitHub Action that runs the full plugin-eval engine across every local plugin and emits a structured report for review — not a PR gate. - scripts/eval_all.py batch-runs EvalEngine.evaluate_plugin across plugins/*/, writing per-plugin JSON plus summary.md and summary.json - Report surfaces composite score, 95% CI, badge, confidence label, anti-pattern flags, and three weakest dimensions per plugin - Separate "Issues requiring attention" section for plugins scoring under 60 or with anti-pattern flags raised - Workflow triggers: workflow_dispatch (choose depth + optional comma-separated plugin filter) and weekly cron (Mondays 06:00 UTC) - Depth quick = static only (no credentials); standard/deep require ANTHROPIC_API_KEY repo secret for LLM judge / Monte Carlo layers - Posts summary.md to the job summary, uploads full eval-reports/ directory as an artifact for 30 days Smoke-tested locally: 77 plugins evaluated at quick depth in < 5s, mean 83.1/100, 2 plugins flagged (incident-response with DEAD_CROSS_REF, plugin-eval with ORPHAN_REFERENCE + DEAD_CROSS_REF).
plugin-eval
Three-layer quality evaluation framework for Claude Code plugins.
Quick Start
cd plugins/plugin-eval
uv sync
# Evaluate a skill (static only, instant)
uv run plugin-eval score path/to/skill --depth quick
# Evaluate with LLM judge (~30s)
uv run plugin-eval score path/to/skill --depth standard
# Full certification (all layers, ~5 min)
uv run plugin-eval certify path/to/skill
Layers
- Static Analysis — Structural checks, anti-pattern detection. Instant, free.
- LLM Judge — Semantic evaluation (triggering, orchestration, output, scope). ~30s, 4 calls.
- Monte Carlo — Statistical reliability via 50–100 simulated runs. ~2–5 min.
Commands
| CLI | Claude Code | Description |
|---|---|---|
plugin-eval score |
/eval |
Score a plugin or skill |
plugin-eval certify |
/certify |
Full certification with badge |
plugin-eval compare |
/compare |
Head-to-head comparison |
plugin-eval init |
— | Build corpus for Elo ranking |
Documentation
See docs/plugin-eval.md for the full reference: layers, dimensions, scoring formula, anti-patterns, statistical methods, and project structure.