feat(plugin-eval): add Claude Code plugin shell (commands, agents, skill)

Add plugin.json, three commands (eval/compare/certify), eval-orchestrator agent,
and evaluation-methodology skill to wire plugin-eval into the claude-agents ecosystem.
This commit is contained in:
Seth Hobson
2026-03-25 17:51:28 -04:00
parent 5849ee3546
commit b4afcc4063
6 changed files with 134 additions and 0 deletions
@@ -0,0 +1,3 @@
{
"name": "plugin-eval"
}
@@ -0,0 +1,34 @@
---
name: eval-orchestrator
description: "Orchestrates plugin quality evaluation. Use PROACTIVELY when evaluating, scoring, or certifying plugin quality."
model: opus
---
You are the PluginEval orchestrator. You coordinate quality evaluation of Claude Code plugins.
## Your Role
When asked to evaluate a plugin or skill:
1. Identify the target path
2. Determine the appropriate depth (quick for CI, standard for dev, deep for certification)
3. Run `plugin-eval` CLI with the right flags
4. Interpret the results and provide actionable recommendations
## Tools Available
Run the CLI:
```bash
cd /Users/wshobson/workspace/claude-agents/plugins/plugin-eval
uv run plugin-eval score <path> --depth <level> --output markdown
```
## Interpreting Results
- **Platinum (90+, Elo 1600+):** Reference quality. Highlight what makes it exceptional.
- **Gold (80+, Elo 1500+):** Production ready. Note 1-2 areas for improvement.
- **Silver (70+, Elo 1400+):** Functional but needs work. Prioritize recommendations.
- **Bronze (60+):** Minimum viable. Significant improvements needed.
- **No badge (<60):** Below standard. Identify critical issues.
Focus recommendations on the lowest-scoring dimensions and any detected anti-patterns.
+15
View File
@@ -0,0 +1,15 @@
---
description: Full quality certification with badge
argument-hint: <path>
---
Run the complete PluginEval certification pipeline (all three layers + Elo ranking) and assign a quality badge.
This takes 15-20 minutes and uses your Max plan for all LLM calls.
## Running
```bash
cd plugins/plugin-eval
uv run plugin-eval certify {argument} --output markdown
```
+13
View File
@@ -0,0 +1,13 @@
---
description: Compare two skills head-to-head
argument-hint: <skill-a> <skill-b>
---
Run a pairwise comparison between two skills and report which is better on each quality dimension.
## Running
```bash
cd plugins/plugin-eval
uv run plugin-eval compare {argument}
```
+27
View File
@@ -0,0 +1,27 @@
---
description: Evaluate a plugin or skill for quality
argument-hint: <path> [--depth quick|standard|deep]
---
Run the PluginEval quality evaluation on a plugin or skill directory.
## Usage
/eval <path> — evaluate at standard depth (Layers 1+2)
/eval <path> --depth quick — static analysis only (instant)
/eval <path> --depth deep — full evaluation with Monte Carlo (15-20 min)
## What It Does
1. Parses the plugin/skill structure (frontmatter, sections, references)
2. Runs static analysis for anti-patterns and structural quality
3. At standard+ depth: LLM judge assesses triggering, orchestration, output quality, scope
4. At deep+ depth: Monte Carlo simulation measures statistical reliability
5. Produces a composite score (0-100), dimension breakdown, and quality badge
## Running
```bash
cd plugins/plugin-eval
uv run plugin-eval score {argument} --depth standard --output markdown
```
@@ -0,0 +1,42 @@
---
name: evaluation-methodology
description: "PluginEval quality methodology — dimensions, rubrics, statistical methods. Use when understanding how plugin quality is measured or when interpreting evaluation results."
---
# Evaluation Methodology
## Quality Dimensions (Ranked by Weight)
1. **Triggering accuracy (0.25)** — Does the skill fire when it should?
2. **Orchestration fitness (0.20)** — Works as a worker in agent→skill hierarchy?
3. **Output quality (0.15)** — Consistent, correct results?
4. **Scope calibration (0.12)** — Right depth and breadth?
5. **Progressive disclosure (0.10)** — SKILL.md lean, depth in refs?
6. **Token efficiency (0.06)** — Minimal context waste?
7. **Robustness (0.05)** — Handles edge cases?
8. **Structural completeness (0.03)** — Right sections present?
9. **Code template quality (0.02)** — Working examples?
10. **Ecosystem coherence (0.02)** — Cross-references, no duplication?
## Three Evaluation Layers
- **Layer 1 (Static):** Deterministic analysis, <2 seconds, no LLM
- **Layer 2 (LLM Judge):** G-Eval style rubric assessment via Agent SDK
- **Layer 3 (Monte Carlo):** N simulated invocations with statistical CIs
## Statistical Methods
- **Wilson score interval:** Triggering activation rate CIs
- **Bootstrap resampling (1000x):** Output quality CIs
- **Clopper-Pearson exact:** Failure rate CIs
- **Cohen's kappa:** Inter-judge agreement
- **Elo/Bradley-Terry:** Pairwise ranking against gold corpus
## Quality Badges
| Badge | Composite | Elo | Meaning |
|-------|-----------|-----|---------|
| Platinum | 90+ | 1600+ | Reference quality |
| Gold | 80+ | 1500+ | Production ready |
| Silver | 70+ | 1400+ | Functional, needs improvement |
| Bronze | 60+ | 1300+ | Minimum viable |