mirror of
https://github.com/wshobson/agents
synced 2026-06-21 14:13:58 +00:00
jon
112197c6bf
The static-layer trigger heuristic was a literal substring match on "Use when …" / "Use this skill when …" / "Use proactively" / "Trigger when …". Three real-world phrasings were missed: 1. Third-person canonical form recommended by Anthropic's own plugin-dev skill-development skill: "This skill should be used when …" (contains "used when", not "use when"). Running the previous regex on plugin-dev itself produces 7 false-positive MISSING_TRIGGER flags. 2. Prepositional temporal triggers: "Use after …", "Use before …", "Use immediately before …", "Use whenever …". Common in self-audit and hook-adjacent skills (e.g. functional-emotions: 6 flags). 3. Path-triggered self-documenting skills: "Auto-loads when …". Extract the pattern to a module-level `_TRIGGER_PATTERN` and share it between the anti-pattern detector and `_description_pushiness` so the two sites stay in sync. Expand it to match `(should be )?use(d)? (this skill )?(immediately )?(when|after|before|whenever)` plus `auto-loads when` and `trigger when`. Verification: - plugin-dev (Anthropic's own): 33.82 → 55.18, MISSING_TRIGGER cleared - functional-emotions: 39.04 → 59.68, all 6 false positives cleared - 22 tests pass, including 11 new positive-form parametrised cases and 2 regression tests against canonical Anthropic phrasings.
plugin-eval
Three-layer quality evaluation framework for Claude Code plugins.
Quick Start
cd plugins/plugin-eval
uv sync
# Evaluate a skill (static only, instant)
uv run plugin-eval score path/to/skill --depth quick
# Evaluate with LLM judge (~30s)
uv run plugin-eval score path/to/skill --depth standard
# Full certification (all layers, ~5 min)
uv run plugin-eval certify path/to/skill
Layers
- Static Analysis — Structural checks, anti-pattern detection. Instant, free.
- LLM Judge — Semantic evaluation (triggering, orchestration, output, scope). ~30s, 4 calls.
- Monte Carlo — Statistical reliability via 50–100 simulated runs. ~2–5 min.
Commands
| CLI | Claude Code | Description |
|---|---|---|
plugin-eval score |
/eval |
Score a plugin or skill |
plugin-eval certify |
/certify |
Full certification with badge |
plugin-eval compare |
/compare |
Head-to-head comparison |
plugin-eval init |
— | Build corpus for Elo ranking |
Documentation
See docs/plugin-eval.md for the full reference: layers, dimensions, scoring formula, anti-patterns, statistical methods, and project structure.