mirror of
https://github.com/wshobson/agents
synced 2026-06-21 14:13:58 +00:00
Seth Hobson
d31e8b8fea
Review feedback:
- codex.py: Codex marketplace entries now conform to the openai/codex
core-plugins schema — source {source: "local", path}, policy
{installation, authentication}, and category (was missing required
fields; flagged P1). Top-level trimmed to {name, plugins}.
- codex.py: interface.shortDescription cuts on a word boundary + ellipsis
instead of a mid-word [:120] slice in committed .codex-plugin manifests.
- cursor.py: top-level .cursor-plugin/plugin.json displayName is now the
short "Claude Code Workflows" instead of the long marketplace description.
Tooling (local drives CI):
- Bump ruff 0.15.14 -> 0.15.15 and ty 0.0.39 -> 0.0.40 in plugin-eval's
uv.lock so CI installs the same versions we run locally, and reformat the
tools/ + plugin-eval sources to ruff 0.15.15 (fixes the failing
ruff format --check; the lock-hash change also busts CI's stale cache).
make validate STRICT=1 clean / garden 0 errors / 442 tests pass; regeneration
remains drift-free.
plugin-eval
Three-layer quality evaluation framework for Claude Code plugins.
Quick Start
cd plugins/plugin-eval
uv sync
# Evaluate a skill (static only, instant)
uv run plugin-eval score path/to/skill --depth quick
# Evaluate with LLM judge (~30s)
uv run plugin-eval score path/to/skill --depth standard
# Full certification (all layers, ~5 min)
uv run plugin-eval certify path/to/skill
Layers
- Static Analysis — Structural checks, anti-pattern detection. Instant, free.
- LLM Judge — Semantic evaluation (triggering, orchestration, output, scope). ~30s, 4 calls.
- Monte Carlo — Statistical reliability via 50–100 simulated runs. ~2–5 min.
Commands
| CLI | Claude Code | Description |
|---|---|---|
plugin-eval score |
/eval |
Score a plugin or skill |
plugin-eval certify |
/certify |
Full certification with badge |
plugin-eval compare |
/compare |
Head-to-head comparison |
plugin-eval init |
— | Build corpus for Elo ranking |
Documentation
See docs/plugin-eval.md for the full reference: layers, dimensions, scoring formula, anti-patterns, statistical methods, and project structure.