mirror of
https://github.com/wshobson/agents
synced 2026-06-21 14:13:58 +00:00
jon
5162141430
* fix(plugin-eval): exempt slash-only and path-triggered skills from MISSING_TRIGGER Skills can be invoked through three mechanisms in Claude Code: 1. Model-driven auto-invocation based on the description (the default, and what MISSING_TRIGGER is designed to gate). 2. User-driven slash invocation only — opt-out via `disable-model-invocation: true` in the SKILL.md frontmatter. 3. Path-triggered auto-load — opt-in via the `paths:` frontmatter glob, where the skill is loaded when the model opens a matching file. The current `MISSING_TRIGGER` check runs against every skill regardless of invocation mechanism. Skills in (2) cannot be auto-invoked at all, so a trigger phrase in the description is irrelevant. Skills in (3) are triggered by file paths, not description matching, so the description serves as documentation rather than as a discovery surface. Add a `_skill_uses_description_trigger(skill)` predicate that consults `skill.frontmatter` and short-circuits the MISSING_TRIGGER check for the two non-description invocation paths. Behaviour for ordinary model-invocable skills is unchanged. Tests: - `test_disable_model_invocation_exempts_skill` — slash-only, no trigger, no flag. - `test_paths_auto_load_exempts_skill` — path-triggered, no trigger, no flag. - `test_disable_model_invocation_false_still_checks_trigger` — explicit `false` does not exempt. - `test_empty_paths_value_still_checks_trigger` — empty string is not a valid auto-load configuration. Full plugin-eval suite (73 tests) passes. * fix: address ruff SIM103/SIM102 and guard paths check against non-string values Simplify conditional returns (SIM103), collapse nested ifs (SIM102), and use isinstance(paths, str) to prevent paths: [] from incorrectly exempting a skill from MISSING_TRIGGER. --------- Co-authored-by: Seth Hobson <wshobson@gmail.com>
plugin-eval
Three-layer quality evaluation framework for Claude Code plugins.
Quick Start
cd plugins/plugin-eval
uv sync
# Evaluate a skill (static only, instant)
uv run plugin-eval score path/to/skill --depth quick
# Evaluate with LLM judge (~30s)
uv run plugin-eval score path/to/skill --depth standard
# Full certification (all layers, ~5 min)
uv run plugin-eval certify path/to/skill
Layers
- Static Analysis — Structural checks, anti-pattern detection. Instant, free.
- LLM Judge — Semantic evaluation (triggering, orchestration, output, scope). ~30s, 4 calls.
- Monte Carlo — Statistical reliability via 50–100 simulated runs. ~2–5 min.
Commands
| CLI | Claude Code | Description |
|---|---|---|
plugin-eval score |
/eval |
Score a plugin or skill |
plugin-eval certify |
/certify |
Full certification with badge |
plugin-eval compare |
/compare |
Head-to-head comparison |
plugin-eval init |
— | Build corpus for Elo ranking |
Documentation
See docs/plugin-eval.md for the full reference: layers, dimensions, scoring formula, anti-patterns, statistical methods, and project structure.