mirror of
https://github.com/wshobson/agents
synced 2026-06-21 14:13:58 +00:00
feat(plugin-eval): add Claude Code plugin shell (commands, agents, skill)
Add plugin.json, three commands (eval/compare/certify), eval-orchestrator agent, and evaluation-methodology skill to wire plugin-eval into the claude-agents ecosystem.
This commit is contained in:
@@ -0,0 +1,3 @@
|
||||
{
|
||||
"name": "plugin-eval"
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
---
|
||||
name: eval-orchestrator
|
||||
description: "Orchestrates plugin quality evaluation. Use PROACTIVELY when evaluating, scoring, or certifying plugin quality."
|
||||
model: opus
|
||||
---
|
||||
|
||||
You are the PluginEval orchestrator. You coordinate quality evaluation of Claude Code plugins.
|
||||
|
||||
## Your Role
|
||||
|
||||
When asked to evaluate a plugin or skill:
|
||||
|
||||
1. Identify the target path
|
||||
2. Determine the appropriate depth (quick for CI, standard for dev, deep for certification)
|
||||
3. Run `plugin-eval` CLI with the right flags
|
||||
4. Interpret the results and provide actionable recommendations
|
||||
|
||||
## Tools Available
|
||||
|
||||
Run the CLI:
|
||||
```bash
|
||||
cd /Users/wshobson/workspace/claude-agents/plugins/plugin-eval
|
||||
uv run plugin-eval score <path> --depth <level> --output markdown
|
||||
```
|
||||
|
||||
## Interpreting Results
|
||||
|
||||
- **Platinum (90+, Elo 1600+):** Reference quality. Highlight what makes it exceptional.
|
||||
- **Gold (80+, Elo 1500+):** Production ready. Note 1-2 areas for improvement.
|
||||
- **Silver (70+, Elo 1400+):** Functional but needs work. Prioritize recommendations.
|
||||
- **Bronze (60+):** Minimum viable. Significant improvements needed.
|
||||
- **No badge (<60):** Below standard. Identify critical issues.
|
||||
|
||||
Focus recommendations on the lowest-scoring dimensions and any detected anti-patterns.
|
||||
@@ -0,0 +1,15 @@
|
||||
---
|
||||
description: Full quality certification with badge
|
||||
argument-hint: <path>
|
||||
---
|
||||
|
||||
Run the complete PluginEval certification pipeline (all three layers + Elo ranking) and assign a quality badge.
|
||||
|
||||
This takes 15-20 minutes and uses your Max plan for all LLM calls.
|
||||
|
||||
## Running
|
||||
|
||||
```bash
|
||||
cd plugins/plugin-eval
|
||||
uv run plugin-eval certify {argument} --output markdown
|
||||
```
|
||||
@@ -0,0 +1,13 @@
|
||||
---
|
||||
description: Compare two skills head-to-head
|
||||
argument-hint: <skill-a> <skill-b>
|
||||
---
|
||||
|
||||
Run a pairwise comparison between two skills and report which is better on each quality dimension.
|
||||
|
||||
## Running
|
||||
|
||||
```bash
|
||||
cd plugins/plugin-eval
|
||||
uv run plugin-eval compare {argument}
|
||||
```
|
||||
@@ -0,0 +1,27 @@
|
||||
---
|
||||
description: Evaluate a plugin or skill for quality
|
||||
argument-hint: <path> [--depth quick|standard|deep]
|
||||
---
|
||||
|
||||
Run the PluginEval quality evaluation on a plugin or skill directory.
|
||||
|
||||
## Usage
|
||||
|
||||
/eval <path> — evaluate at standard depth (Layers 1+2)
|
||||
/eval <path> --depth quick — static analysis only (instant)
|
||||
/eval <path> --depth deep — full evaluation with Monte Carlo (15-20 min)
|
||||
|
||||
## What It Does
|
||||
|
||||
1. Parses the plugin/skill structure (frontmatter, sections, references)
|
||||
2. Runs static analysis for anti-patterns and structural quality
|
||||
3. At standard+ depth: LLM judge assesses triggering, orchestration, output quality, scope
|
||||
4. At deep+ depth: Monte Carlo simulation measures statistical reliability
|
||||
5. Produces a composite score (0-100), dimension breakdown, and quality badge
|
||||
|
||||
## Running
|
||||
|
||||
```bash
|
||||
cd plugins/plugin-eval
|
||||
uv run plugin-eval score {argument} --depth standard --output markdown
|
||||
```
|
||||
@@ -0,0 +1,42 @@
|
||||
---
|
||||
name: evaluation-methodology
|
||||
description: "PluginEval quality methodology — dimensions, rubrics, statistical methods. Use when understanding how plugin quality is measured or when interpreting evaluation results."
|
||||
---
|
||||
|
||||
# Evaluation Methodology
|
||||
|
||||
## Quality Dimensions (Ranked by Weight)
|
||||
|
||||
1. **Triggering accuracy (0.25)** — Does the skill fire when it should?
|
||||
2. **Orchestration fitness (0.20)** — Works as a worker in agent→skill hierarchy?
|
||||
3. **Output quality (0.15)** — Consistent, correct results?
|
||||
4. **Scope calibration (0.12)** — Right depth and breadth?
|
||||
5. **Progressive disclosure (0.10)** — SKILL.md lean, depth in refs?
|
||||
6. **Token efficiency (0.06)** — Minimal context waste?
|
||||
7. **Robustness (0.05)** — Handles edge cases?
|
||||
8. **Structural completeness (0.03)** — Right sections present?
|
||||
9. **Code template quality (0.02)** — Working examples?
|
||||
10. **Ecosystem coherence (0.02)** — Cross-references, no duplication?
|
||||
|
||||
## Three Evaluation Layers
|
||||
|
||||
- **Layer 1 (Static):** Deterministic analysis, <2 seconds, no LLM
|
||||
- **Layer 2 (LLM Judge):** G-Eval style rubric assessment via Agent SDK
|
||||
- **Layer 3 (Monte Carlo):** N simulated invocations with statistical CIs
|
||||
|
||||
## Statistical Methods
|
||||
|
||||
- **Wilson score interval:** Triggering activation rate CIs
|
||||
- **Bootstrap resampling (1000x):** Output quality CIs
|
||||
- **Clopper-Pearson exact:** Failure rate CIs
|
||||
- **Cohen's kappa:** Inter-judge agreement
|
||||
- **Elo/Bradley-Terry:** Pairwise ranking against gold corpus
|
||||
|
||||
## Quality Badges
|
||||
|
||||
| Badge | Composite | Elo | Meaning |
|
||||
|-------|-----------|-----|---------|
|
||||
| Platinum | 90+ | 1600+ | Reference quality |
|
||||
| Gold | 80+ | 1500+ | Production ready |
|
||||
| Silver | 70+ | 1400+ | Functional, needs improvement |
|
||||
| Bronze | 60+ | 1300+ | Minimum viable |
|
||||
Reference in New Issue
Block a user