mirror of
https://github.com/wshobson/agents
synced 2026-06-21 14:13:58 +00:00
docs(plugin-eval): add CLAUDE.md, full evaluation framework docs, and README updates
- Create CLAUDE.md with project conventions and PluginEval quick reference - Add docs/plugin-eval.md with complete framework documentation (layers, dimensions, scoring, anti-patterns, statistical methods, project structure) - Add PluginEval section to main README under What's New - Expand plugin-eval/README.md with quick start and command table
This commit is contained in:
Seth Hobson
@@ -0,0 +1,153 @@
|
||||
# Project: claude-agents
|
||||
|
||||
Claude Code plugin marketplace — 72 focused plugins, 112 agents, 146 skills, 79 tools.
|
||||
|
||||
## Repository Structure
|
||||
|
||||
```
|
||||
claude-agents/
|
||||
├── .claude-plugin/marketplace.json # Registry of all plugins
|
||||
├── plugins/ # All 72 plugins
|
||||
│ ├── <plugin-name>/
|
||||
│ │ ├── .claude-plugin/plugin.json
|
||||
│ │ ├── agents/*.md
|
||||
│ │ ├── commands/*.md
|
||||
│ │ └── skills/<skill-name>/SKILL.md
|
||||
│ └── ...
|
||||
├── docs/ # Documentation
|
||||
│ ├── plugins.md # Plugin catalog
|
||||
│ ├── agents.md # Agent reference
|
||||
│ ├── agent-skills.md # Skills reference
|
||||
│ ├── usage.md # Usage guide
|
||||
│ ├── architecture.md # Design principles
|
||||
│ └── plugin-eval.md # Evaluation framework
|
||||
└── tools/ # Development utilities
|
||||
```
|
||||
|
||||
## Plugin Authoring Conventions
|
||||
|
||||
### Agent frontmatter
|
||||
|
||||
```yaml
|
||||
---
|
||||
name: agent-name
|
||||
description: "What this agent does. Use PROACTIVELY when [trigger conditions]."
|
||||
model: opus|sonnet|haiku|inherit
|
||||
color: blue|green|red|yellow|cyan|magenta # optional
|
||||
tools: Read, Grep, Glob # optional — restricts available tools
|
||||
---
|
||||
```
|
||||
|
||||
### Skill structure
|
||||
|
||||
```
|
||||
skills/<skill-name>/
|
||||
├── SKILL.md # Required — frontmatter + content
|
||||
├── references/ # Optional — supporting material
|
||||
│ └── *.md
|
||||
└── assets/ # Optional — templates, configs
|
||||
```
|
||||
|
||||
Skill frontmatter:
|
||||
|
||||
```yaml
|
||||
---
|
||||
name: skill-name
|
||||
description: "Use this skill when [specific trigger conditions]."
|
||||
---
|
||||
```
|
||||
|
||||
### Command frontmatter
|
||||
|
||||
```yaml
|
||||
---
|
||||
description: What this command does
|
||||
argument-hint: <path> [--flag]
|
||||
---
|
||||
```
|
||||
|
||||
### plugin.json
|
||||
|
||||
Only `name` is required. Agents, commands, and skills are auto-discovered from directory structure.
|
||||
|
||||
```json
|
||||
{ "name": "plugin-name" }
|
||||
```
|
||||
|
||||
### marketplace.json
|
||||
|
||||
Lists all plugin component paths for the registry. Agents as `./agents/name.md`, skills as `./skills/skill-name` (directory, not SKILL.md).
|
||||
|
||||
## Model Tiers
|
||||
|
||||
| Tier | Model | Use Case |
|
||||
| ------ | ------- | ------------------------------------------------------ |
|
||||
| Tier 1 | Opus | Architecture, security, code review, production coding |
|
||||
| Tier 2 | Inherit | Complex tasks — user chooses model |
|
||||
| Tier 3 | Sonnet | Docs, testing, debugging, support |
|
||||
| Tier 4 | Haiku | Fast ops, SEO, deployment, simple tasks |
|
||||
|
||||
## PluginEval — Quality Evaluation Framework
|
||||
|
||||
Three-layer evaluation system in `plugins/plugin-eval/`. Full docs: [docs/plugin-eval.md](docs/plugin-eval.md).
|
||||
|
||||
### Quick Reference
|
||||
|
||||
```bash
|
||||
cd plugins/plugin-eval
|
||||
|
||||
# Run tests
|
||||
uv run pytest
|
||||
|
||||
# Evaluate a skill (static only)
|
||||
uv run plugin-eval score path/to/skill --depth quick --output json
|
||||
|
||||
# Evaluate with LLM judge
|
||||
uv run plugin-eval score path/to/skill --depth standard
|
||||
|
||||
# Full certification (all 3 layers)
|
||||
uv run plugin-eval certify path/to/skill
|
||||
|
||||
# Compare two skills
|
||||
uv run plugin-eval compare path/to/skill-a path/to/skill-b
|
||||
|
||||
# Build corpus for Elo ranking
|
||||
uv run plugin-eval init plugins/
|
||||
```
|
||||
|
||||
### Evaluation Layers
|
||||
|
||||
1. **Static** (Layer 1) — Deterministic structural analysis. < 2 seconds, free, always runs.
|
||||
2. **LLM Judge** (Layer 2) — Semantic evaluation via Claude (Haiku + Sonnet). ~30s, 4 LLM calls.
|
||||
3. **Monte Carlo** (Layer 3) — Statistical reliability via N simulated runs. ~2–5 min, 50–100 calls.
|
||||
|
||||
### 10 Dimensions (weights)
|
||||
|
||||
triggering_accuracy (25%), orchestration_fitness (20%), output_quality (15%), scope_calibration (12%), progressive_disclosure (10%), token_efficiency (6%), robustness (5%), structural_completeness (3%), code_template_quality (2%), ecosystem_coherence (2%)
|
||||
|
||||
### Badges
|
||||
|
||||
Platinum ≥90, Gold ≥80, Silver ≥70, Bronze ≥60
|
||||
|
||||
### Anti-Patterns
|
||||
|
||||
OVER_CONSTRAINED (>15 MUST/ALWAYS/NEVER), EMPTY_DESCRIPTION (<20 chars), MISSING_TRIGGER (no "Use when…"), BLOATED_SKILL (>800 lines no refs), ORPHAN_REFERENCE (dead link), DEAD_CROSS_REF (missing skill)
|
||||
|
||||
### Tech Stack
|
||||
|
||||
- Python ≥ 3.12, uv, ruff, ty, pytest
|
||||
- Dependencies: pydantic, typer, rich, pyyaml
|
||||
- Optional: claude-agent-sdk (LLM layers), anthropic (API alternative)
|
||||
|
||||
## Development
|
||||
|
||||
### Python Tooling
|
||||
|
||||
Use the Astral Rust toolchain: `uv` (package manager), `ruff` (linter/formatter), `ty` (type checker). Do not use pip, mypy, or black.
|
||||
|
||||
### Adding a Plugin
|
||||
|
||||
1. Create `plugins/<name>/` with `.claude-plugin/plugin.json`
|
||||
2. Add agents in `agents/`, commands in `commands/`, skills in `skills/`
|
||||
3. Update `.claude-plugin/marketplace.json`
|
||||
4. Follow naming conventions: lowercase, hyphen-separated
|
||||
@@ -119,6 +119,7 @@ rm -rf ~/.claude/plugins/cache/claude-code-workflows && rm ~/.claude/plugins/ins
|
||||
- **[Agent Skills](docs/agent-skills.md)** - 146 specialized skills with progressive disclosure
|
||||
- **[Usage Guide](docs/usage.md)** - Commands, workflows, and best practices
|
||||
- **[Architecture](docs/architecture.md)** - Design principles and patterns
|
||||
- **[PluginEval](docs/plugin-eval.md)** - Quality evaluation framework (layers, dimensions, scoring)
|
||||
|
||||
### Quick Links
|
||||
|
||||
@@ -130,7 +131,36 @@ rm -rf ~/.claude/plugins/cache/claude-code-workflows && rm ~/.claude/plugins/ins
|
||||
|
||||
## What's New
|
||||
|
||||
### Agent Teams Plugin (NEW)
|
||||
### PluginEval — Quality Evaluation Framework (NEW)
|
||||
|
||||
A three-layer evaluation framework for measuring and certifying plugin/skill quality:
|
||||
|
||||
```bash
|
||||
/plugin install plugin-eval@claude-code-workflows
|
||||
```
|
||||
|
||||
- **Three Evaluation Layers** — Static analysis (instant), LLM judge (semantic), Monte Carlo simulation (statistical)
|
||||
- **10 Quality Dimensions** — Triggering accuracy, orchestration fitness, output quality, scope calibration, progressive disclosure, token efficiency, robustness, structural completeness, code template quality, ecosystem coherence
|
||||
- **Quality Badges** — Platinum (★★★★★), Gold (★★★★), Silver (★★★), Bronze (★★)
|
||||
- **Anti-Pattern Detection** — OVER_CONSTRAINED, EMPTY_DESCRIPTION, MISSING_TRIGGER, BLOATED_SKILL, ORPHAN_REFERENCE, DEAD_CROSS_REF
|
||||
- **Statistical Rigor** — Wilson score CI, bootstrap CI, Clopper-Pearson exact CI, Elo ranking
|
||||
- **CLI + Claude Code** — `uv run plugin-eval score/certify/compare` or `/eval`, `/certify`, `/compare` commands
|
||||
- **CI Gate** — `--threshold` flag exits non-zero below a minimum score
|
||||
|
||||
```bash
|
||||
# Quick evaluation (static only, instant)
|
||||
uv run plugin-eval score path/to/skill --depth quick
|
||||
|
||||
# Standard evaluation (static + LLM judge)
|
||||
uv run plugin-eval score path/to/skill --depth standard
|
||||
|
||||
# Full certification (all layers + Elo)
|
||||
uv run plugin-eval certify path/to/skill
|
||||
```
|
||||
|
||||
[→ View PluginEval documentation](docs/plugin-eval.md)
|
||||
|
||||
### Agent Teams Plugin
|
||||
|
||||
Orchestrate multi-agent teams for parallel workflows using Claude Code's experimental Agent Teams feature:
|
||||
|
||||
|
||||
@@ -0,0 +1,433 @@
|
||||
# PluginEval: Quality Evaluation Framework
|
||||
|
||||
PluginEval is a three-layer quality evaluation framework for Claude Code plugins and skills. It combines deterministic static analysis, LLM-based semantic judging, and Monte Carlo simulation to produce calibrated quality scores with confidence intervals.
|
||||
|
||||
## Overview
|
||||
|
||||
PluginEval answers the question: **"How good is this plugin or skill?"** It evaluates across 10 quality dimensions, detects anti-patterns, assigns letter grades, and awards quality badges (Bronze through Platinum).
|
||||
|
||||
### Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────┐
|
||||
│ CLI / Commands │
|
||||
│ score · certify · compare · init │
|
||||
├─────────────────────────────────────────────────┤
|
||||
│ Eval Engine │
|
||||
│ Composite scoring, layer blending │
|
||||
├────────────┬────────────────┬───────────────────┤
|
||||
│ Layer 1 │ Layer 2 │ Layer 3 │
|
||||
│ Static │ LLM Judge │ Monte Carlo │
|
||||
│ Analysis │ (Semantic) │ (Statistical) │
|
||||
│ <2s, free │ ~30s, 4 calls │ ~2min, 50 calls │
|
||||
├────────────┴────────────────┴───────────────────┤
|
||||
│ Parser Layer │
|
||||
│ SKILL.md, agents/*.md, plugin.json │
|
||||
├─────────────────────────────────────────────────┤
|
||||
│ Statistical Methods │
|
||||
│ Wilson CI · Bootstrap CI · Clopper-Pearson │
|
||||
│ Cohen's κ · Coefficient of Variation │
|
||||
├─────────────────────────────────────────────────┤
|
||||
│ Corpus & Elo Ranking │
|
||||
│ Gold standard index · Pairwise comparison │
|
||||
└─────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Installation & Setup
|
||||
|
||||
PluginEval lives in `plugins/plugin-eval/` and uses [uv](https://docs.astral.sh/uv/) for dependency management.
|
||||
|
||||
```bash
|
||||
cd plugins/plugin-eval
|
||||
|
||||
# Install core dependencies (static analysis only)
|
||||
uv sync
|
||||
|
||||
# Install with LLM support (Layers 2 & 3)
|
||||
uv sync --extra llm
|
||||
|
||||
# Install with direct API support
|
||||
uv sync --extra api
|
||||
|
||||
# Install dev dependencies (tests, linting)
|
||||
uv sync --extra dev
|
||||
```
|
||||
|
||||
### Requirements
|
||||
|
||||
- Python ≥ 3.12
|
||||
- Core: `pydantic`, `typer`, `rich`, `pyyaml`
|
||||
- LLM layers: `claude-agent-sdk` (uses Claude Code Max plan by default)
|
||||
- API alternative: `anthropic` SDK (requires `ANTHROPIC_API_KEY`)
|
||||
|
||||
## CLI Commands
|
||||
|
||||
### `score` — Evaluate a plugin or skill
|
||||
|
||||
```bash
|
||||
# Quick evaluation (static only, instant)
|
||||
uv run plugin-eval score path/to/skill --depth quick
|
||||
|
||||
# Standard evaluation (static + LLM judge)
|
||||
uv run plugin-eval score path/to/skill --depth standard
|
||||
|
||||
# Deep evaluation (all three layers)
|
||||
uv run plugin-eval score path/to/skill --depth deep
|
||||
|
||||
# Output formats
|
||||
uv run plugin-eval score path/to/skill --output json
|
||||
uv run plugin-eval score path/to/skill --output markdown
|
||||
uv run plugin-eval score path/to/skill --output html
|
||||
|
||||
# CI gate: exit code 1 if below threshold
|
||||
uv run plugin-eval score path/to/skill --threshold 70
|
||||
```
|
||||
|
||||
**Options:**
|
||||
| Option | Default | Description |
|
||||
|--------|---------|-------------|
|
||||
| `--depth` | `standard` | `quick`, `standard`, `deep`, `thorough` |
|
||||
| `--output` | `markdown` | `json`, `markdown`, `html` |
|
||||
| `--verbose` | `false` | Show detailed output |
|
||||
| `--concurrency` | `4` | Max concurrent LLM calls (1–20) |
|
||||
| `--auth` | `max` | Auth mode: `max` (Claude Code Max plan) or `api-key` |
|
||||
| `--threshold` | none | Minimum score; exit 1 if below |
|
||||
|
||||
### `certify` — Full certification with badge
|
||||
|
||||
Runs at `deep` depth (all three layers). Takes 15–20 minutes.
|
||||
|
||||
```bash
|
||||
uv run plugin-eval certify path/to/skill --output markdown
|
||||
```
|
||||
|
||||
### `compare` — Head-to-head comparison
|
||||
|
||||
Compare two skills side-by-side across all dimensions.
|
||||
|
||||
```bash
|
||||
uv run plugin-eval compare path/to/skill-a path/to/skill-b
|
||||
```
|
||||
|
||||
### `init` — Initialize corpus
|
||||
|
||||
Build a gold-standard corpus index from a plugins directory for Elo ranking.
|
||||
|
||||
```bash
|
||||
uv run plugin-eval init plugins/ --corpus-dir ~/.plugineval/corpus
|
||||
```
|
||||
|
||||
## Claude Code Integration
|
||||
|
||||
PluginEval is also a Claude Code plugin with agents and commands.
|
||||
|
||||
### Slash Commands
|
||||
|
||||
| Command | Description |
|
||||
| ------------------ | -------------------------------------------------------- |
|
||||
| `/eval <path>` | Evaluate a plugin or skill (orchestrates static + judge) |
|
||||
| `/certify <path>` | Full certification pipeline with badge |
|
||||
| `/compare <a> <b>` | Head-to-head skill comparison |
|
||||
|
||||
### Agents
|
||||
|
||||
| Agent | Model | Role |
|
||||
| ------------------- | ------ | ---------------------------------------------------------------------- |
|
||||
| `eval-orchestrator` | Opus | Coordinates evaluation: runs CLI, dispatches judge, computes composite |
|
||||
| `eval-judge` | Sonnet | LLM judge: scores 4 semantic dimensions with anchored rubrics |
|
||||
|
||||
### Skill
|
||||
|
||||
The `evaluation-methodology` skill provides the full scoring methodology reference, including dimension definitions, rubric anchors, blend weights, and improvement guidance.
|
||||
|
||||
## The Three Evaluation Layers
|
||||
|
||||
### Layer 1: Static Analysis
|
||||
|
||||
**Speed:** < 2 seconds. **Cost:** Free (no LLM calls). **Deterministic.**
|
||||
|
||||
Runs six structural sub-checks against the parsed SKILL.md:
|
||||
|
||||
| Sub-check | Weight | What it measures |
|
||||
| ------------------------- | ------ | --------------------------------------------------------------------------------- |
|
||||
| `frontmatter_quality` | 35% | Name, description length, trigger-phrase quality ("Use when…", "Use PROACTIVELY") |
|
||||
| `orchestration_wiring` | 25% | Output/input documentation, code examples, orchestrator anti-pattern |
|
||||
| `progressive_disclosure` | 15% | Line count vs. sweet spot (200–600 lines), references/ and assets/ directories |
|
||||
| `structural_completeness` | 10% | Heading density, code blocks, examples section, troubleshooting section |
|
||||
| `token_efficiency` | 10% | MUST/NEVER/ALWAYS density, duplicate-line detection |
|
||||
| `ecosystem_coherence` | 5% | Cross-references to other skills/agents, "related"/"see also" mentions |
|
||||
|
||||
Also detects anti-patterns (see below) and applies a multiplicative penalty.
|
||||
|
||||
### Layer 2: LLM Judge
|
||||
|
||||
**Speed:** ~30 seconds. **Cost:** 4 LLM calls (Haiku + Sonnet). **Requires `claude-agent-sdk`.**
|
||||
|
||||
Uses Claude as a semantic evaluator across 4 dimensions with anchored rubrics:
|
||||
|
||||
| Dimension | Model | Method |
|
||||
| ----------------------- | ------ | ---------------------------------------------------------------------------- |
|
||||
| `triggering_accuracy` | Haiku | Generates 10 synthetic prompts (5 should-trigger, 5 should-not), computes F1 |
|
||||
| `orchestration_fitness` | Sonnet | Rates worker-vs-orchestrator role using 5-point anchored rubric |
|
||||
| `output_quality` | Sonnet | Simulates 3 realistic tasks, evaluates expected output quality |
|
||||
| `scope_calibration` | Sonnet | Rates scope appropriateness using 5-point anchored rubric |
|
||||
|
||||
All 4 assessments run concurrently with semaphore-based throttling.
|
||||
|
||||
### Layer 3: Monte Carlo Simulation
|
||||
|
||||
**Speed:** ~2 minutes (50 runs) to ~5 minutes (100 runs). **Cost:** 50–100 LLM calls. **Requires `claude-agent-sdk`.**
|
||||
|
||||
Generates 15 varied prompts via Haiku, then runs N simulations to compute statistical reliability:
|
||||
|
||||
| Metric | Measure | Statistical Method |
|
||||
| ------------------ | --------------------------------------- | --------------------------------- |
|
||||
| Activation rate | % of runs where skill activated | Wilson score CI |
|
||||
| Output consistency | Mean quality + coefficient of variation | Bootstrap CI (1000 resamples) |
|
||||
| Failure rate | % of runs that errored | Clopper-Pearson exact CI |
|
||||
| Token efficiency | Median tokens, IQR, outlier detection | Normalized against 8000-token cap |
|
||||
|
||||
## Evaluation Depths
|
||||
|
||||
| Depth | Layers | Confidence Label | Time | Cost |
|
||||
| ---------- | --------------------------------------- | ---------------- | ------ | -------------- |
|
||||
| `quick` | Static only | Estimated | < 2s | Free |
|
||||
| `standard` | Static + Judge | Assessed | ~30s | 4 LLM calls |
|
||||
| `deep` | Static + Judge + Monte Carlo (50 runs) | Certified | ~3 min | ~54 LLM calls |
|
||||
| `thorough` | Static + Judge + Monte Carlo (100 runs) | Certified+ | ~6 min | ~104 LLM calls |
|
||||
|
||||
## The 10 Quality Dimensions
|
||||
|
||||
Each dimension has a weight and receives scores from different layers, blended using per-dimension weights:
|
||||
|
||||
| Dimension | Weight | Static | Judge | Monte Carlo | What it measures |
|
||||
| ------------------------- | ------ | ------ | ----- | ----------- | ------------------------------------------------ |
|
||||
| `triggering_accuracy` | 25% | 0.15 | 0.25 | 0.60 | Does the description fire for the right prompts? |
|
||||
| `orchestration_fitness` | 20% | 0.10 | 0.70 | 0.20 | Is it a composable worker, not an orchestrator? |
|
||||
| `output_quality` | 15% | 0.00 | 0.40 | 0.60 | Would it produce correct, useful output? |
|
||||
| `scope_calibration` | 12% | 0.30 | 0.55 | 0.15 | Is the scope well-sized for its domain? |
|
||||
| `progressive_disclosure` | 10% | 0.80 | 0.20 | 0.00 | Does it use references/ for large content? |
|
||||
| `token_efficiency` | 6% | 0.40 | 0.10 | 0.50 | Is it concise without repetition? |
|
||||
| `robustness` | 5% | 0.00 | 0.20 | 0.80 | Does it handle varied inputs reliably? |
|
||||
| `structural_completeness` | 3% | 0.90 | 0.10 | 0.00 | Does it have headings, code, examples? |
|
||||
| `code_template_quality` | 2% | 0.30 | 0.70 | 0.00 | Are code examples production-ready? |
|
||||
| `ecosystem_coherence` | 2% | 0.85 | 0.15 | 0.00 | Does it link to related skills/agents? |
|
||||
|
||||
### Composite Score Formula
|
||||
|
||||
```
|
||||
Final = Σ(dimension_weight × blended_score) × 100 × anti_pattern_penalty
|
||||
```
|
||||
|
||||
Where `blended_score` for each dimension is a weighted combination of available layer scores, renormalized to the layers actually present.
|
||||
|
||||
## Quality Badges
|
||||
|
||||
| Badge | Score | Elo | Stars | Meaning |
|
||||
| -------- | ----- | ------ | ----- | ------------------------ |
|
||||
| Platinum | ≥ 90 | ≥ 1600 | ★★★★★ | Reference quality |
|
||||
| Gold | ≥ 80 | ≥ 1500 | ★★★★ | Production ready |
|
||||
| Silver | ≥ 70 | ≥ 1400 | ★★★ | Functional, needs polish |
|
||||
| Bronze | ≥ 60 | ≥ 1300 | ★★ | Minimum viable |
|
||||
|
||||
Badges require both score AND Elo thresholds when Elo data is available.
|
||||
|
||||
## Letter Grades
|
||||
|
||||
Scores are also converted to letter grades:
|
||||
|
||||
| Grade | Score Range |
|
||||
| ----- | ----------- |
|
||||
| A+ | ≥ 97 |
|
||||
| A | ≥ 93 |
|
||||
| A- | ≥ 90 |
|
||||
| B+ | ≥ 87 |
|
||||
| B | ≥ 83 |
|
||||
| B- | ≥ 80 |
|
||||
| C+ | ≥ 77 |
|
||||
| C | ≥ 73 |
|
||||
| C- | ≥ 70 |
|
||||
| D+ | ≥ 67 |
|
||||
| D | ≥ 63 |
|
||||
| D- | ≥ 60 |
|
||||
| F | < 60 |
|
||||
|
||||
## Anti-Pattern Detection
|
||||
|
||||
The static analyzer detects these anti-patterns, each with a severity that contributes to a multiplicative penalty:
|
||||
|
||||
| Flag | Severity | Trigger |
|
||||
| ------------------- | -------- | --------------------------------------------- |
|
||||
| `OVER_CONSTRAINED` | 10% | > 15 MUST/ALWAYS/NEVER directives |
|
||||
| `EMPTY_DESCRIPTION` | 10% | Description < 20 characters |
|
||||
| `MISSING_TRIGGER` | 15% | No "Use when…" trigger phrase in description |
|
||||
| `BLOATED_SKILL` | 10% | > 800 lines without a references/ directory |
|
||||
| `ORPHAN_REFERENCE` | 5% | Dead link to a file in references/ |
|
||||
| `DEAD_CROSS_REF` | 5% | Cross-reference to a non-existent skill/agent |
|
||||
|
||||
**Penalty formula:** `penalty = max(0.5, 1.0 − 0.05 × count)` — each anti-pattern reduces the score by 5%, flooring at 50%.
|
||||
|
||||
## Elo Ranking System
|
||||
|
||||
For relative quality comparison against a corpus of known skills:
|
||||
|
||||
- **Initial rating:** 1500
|
||||
- **K-factor:** 32
|
||||
- **Confidence intervals:** Bootstrap resampling (500 resamples)
|
||||
- **Corpus management:** `init` command indexes all skills from a plugins directory
|
||||
- **Reference selection:** Matches by category and similar line count
|
||||
|
||||
The Elo system uses the standard formula: `E(A) = 1 / (1 + 10^((Rb - Ra) / 400))`.
|
||||
|
||||
## Corpus Management
|
||||
|
||||
The corpus is a JSON index of all skills used for Elo comparisons:
|
||||
|
||||
```bash
|
||||
# Build corpus from your plugins directory
|
||||
uv run plugin-eval init plugins/ --corpus-dir ~/.plugineval/corpus
|
||||
|
||||
# The corpus stores:
|
||||
# - Skill name, path, category, line count
|
||||
# - Current Elo rating (updated after each comparison)
|
||||
```
|
||||
|
||||
Reference skills are selected by matching category and approximate line count.
|
||||
|
||||
## Statistical Methods
|
||||
|
||||
PluginEval uses rigorous statistical methods throughout:
|
||||
|
||||
| Method | Used For | Details |
|
||||
| ------------------------ | -------------------------- | ----------------------------------------- |
|
||||
| Wilson score CI | Activation rate confidence | Handles small-sample binomial proportions |
|
||||
| Bootstrap CI | Output quality confidence | 1000 resamples, percentile method |
|
||||
| Clopper-Pearson | Failure rate confidence | Exact CI for small failure counts |
|
||||
| Coefficient of variation | Output consistency | std/mean ratio; lower = more consistent |
|
||||
| Cohen's kappa | Inter-rater agreement | For multi-judge scenarios |
|
||||
|
||||
All statistical functions are pure Python with no external dependencies (no scipy/numpy required).
|
||||
|
||||
## Parser
|
||||
|
||||
The parser extracts structured data from Claude Code plugin files:
|
||||
|
||||
- **Skills:** Parses SKILL.md frontmatter (name, description), counts headings, code blocks, languages, MUST/NEVER/ALWAYS directives, cross-references, and detects references/ and assets/ directories
|
||||
- **Agents:** Parses agent .md frontmatter (name, description, model, tools), detects proactive triggers and skill references
|
||||
- **Plugins:** Aggregates all skills and agents from a plugin directory
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
plugins/plugin-eval/
|
||||
├── .claude-plugin/
|
||||
│ └── plugin.json # Claude Code plugin manifest
|
||||
├── agents/
|
||||
│ ├── eval-orchestrator.md # Orchestrates evaluation (Opus)
|
||||
│ └── eval-judge.md # LLM judge agent (Sonnet)
|
||||
├── commands/
|
||||
│ ├── eval.md # /eval slash command
|
||||
│ ├── certify.md # /certify slash command
|
||||
│ └── compare.md # /compare slash command
|
||||
├── skills/
|
||||
│ └── evaluation-methodology/
|
||||
│ ├── SKILL.md # Full methodology reference
|
||||
│ └── references/
|
||||
│ └── rubrics.md # Detailed rubric anchors
|
||||
├── src/plugin_eval/
|
||||
│ ├── __init__.py
|
||||
│ ├── cli.py # Typer CLI (score, certify, compare, init)
|
||||
│ ├── engine.py # Eval engine (layer coordination, composite scoring)
|
||||
│ ├── models.py # Pydantic models (Depth, Badge, EvalConfig, results)
|
||||
│ ├── parser.py # Plugin/skill/agent parser
|
||||
│ ├── reporter.py # JSON/Markdown/HTML output
|
||||
│ ├── corpus.py # Gold standard corpus for Elo ranking
|
||||
│ ├── elo.py # Elo rating calculator with bootstrap CI
|
||||
│ ├── stats.py # Statistical methods (Wilson, bootstrap, Clopper-Pearson)
|
||||
│ └── layers/
|
||||
│ ├── __init__.py
|
||||
│ ├── static.py # Layer 1: deterministic structural analysis
|
||||
│ ├── judge.py # Layer 2: LLM semantic evaluation
|
||||
│ └── monte_carlo.py # Layer 3: statistical reliability simulation
|
||||
├── tests/ # Comprehensive test suite
|
||||
│ ├── conftest.py
|
||||
│ ├── test_cli.py
|
||||
│ ├── test_engine.py
|
||||
│ ├── test_static.py
|
||||
│ ├── test_judge.py
|
||||
│ ├── test_monte_carlo.py
|
||||
│ ├── test_models.py
|
||||
│ ├── test_parser.py
|
||||
│ ├── test_reporter.py
|
||||
│ ├── test_corpus.py
|
||||
│ ├── test_elo.py
|
||||
│ ├── test_stats.py
|
||||
│ └── test_e2e.py # End-to-end tests against real plugins
|
||||
├── pyproject.toml # uv/hatch project config
|
||||
└── uv.lock
|
||||
```
|
||||
|
||||
## Running Tests
|
||||
|
||||
```bash
|
||||
cd plugins/plugin-eval
|
||||
|
||||
# Run all tests
|
||||
uv run pytest
|
||||
|
||||
# Run with coverage
|
||||
uv run pytest --cov=plugin_eval
|
||||
|
||||
# Run specific test file
|
||||
uv run pytest tests/test_static.py
|
||||
|
||||
# Run e2e tests (requires real plugin corpus)
|
||||
uv run pytest tests/test_e2e.py
|
||||
```
|
||||
|
||||
## Example Output
|
||||
|
||||
### Markdown Report
|
||||
|
||||
```
|
||||
# PluginEval Report
|
||||
|
||||
**Path:** `plugins/python-development/skills/async-python-patterns`
|
||||
**Timestamp:** 2025-03-26T12:00:00+00:00
|
||||
**Depth:** standard
|
||||
|
||||
## Overall Score
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| Score | **78.3/100** |
|
||||
| Confidence | Assessed |
|
||||
| Badge | Silver |
|
||||
|
||||
## Layer Breakdown
|
||||
|
||||
| Layer | Score | Anti-Patterns |
|
||||
|-------|-------|---------------|
|
||||
| static | 0.742 | 0 |
|
||||
| judge | 0.811 | 0 |
|
||||
|
||||
## Dimension Scores
|
||||
|
||||
| Dimension | Weight | Score | Grade |
|
||||
|-----------|--------|-------|-------|
|
||||
| Triggering Accuracy | 25% | 0.850 | B |
|
||||
| Orchestration Fitness | 20% | 0.780 | C+ |
|
||||
| Output Quality | 15% | 0.820 | B- |
|
||||
| Scope Calibration | 12% | 0.750 | C |
|
||||
| Progressive Disclosure | 10% | 0.600 | D- |
|
||||
| Token Efficiency | 6% | 0.910 | A- |
|
||||
| ...
|
||||
```
|
||||
|
||||
## Tooling
|
||||
|
||||
- **Package manager:** [uv](https://docs.astral.sh/uv/)
|
||||
- **Linter/formatter:** [ruff](https://docs.astral.sh/ruff/) (target Python 3.12, line length 100)
|
||||
- **Type checker:** [ty](https://docs.astral.sh/ty/)
|
||||
- **Test framework:** pytest with pytest-asyncio
|
||||
- **Build system:** hatchling
|
||||
@@ -1,3 +1,38 @@
|
||||
# plugin-eval
|
||||
|
||||
Three-layer quality evaluation framework for Claude Code plugins.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
cd plugins/plugin-eval
|
||||
uv sync
|
||||
|
||||
# Evaluate a skill (static only, instant)
|
||||
uv run plugin-eval score path/to/skill --depth quick
|
||||
|
||||
# Evaluate with LLM judge (~30s)
|
||||
uv run plugin-eval score path/to/skill --depth standard
|
||||
|
||||
# Full certification (all layers, ~5 min)
|
||||
uv run plugin-eval certify path/to/skill
|
||||
```
|
||||
|
||||
## Layers
|
||||
|
||||
1. **Static Analysis** — Structural checks, anti-pattern detection. Instant, free.
|
||||
2. **LLM Judge** — Semantic evaluation (triggering, orchestration, output, scope). ~30s, 4 calls.
|
||||
3. **Monte Carlo** — Statistical reliability via 50–100 simulated runs. ~2–5 min.
|
||||
|
||||
## Commands
|
||||
|
||||
| CLI | Claude Code | Description |
|
||||
| --------------------- | ----------- | ----------------------------- |
|
||||
| `plugin-eval score` | `/eval` | Score a plugin or skill |
|
||||
| `plugin-eval certify` | `/certify` | Full certification with badge |
|
||||
| `plugin-eval compare` | `/compare` | Head-to-head comparison |
|
||||
| `plugin-eval init` | — | Build corpus for Elo ranking |
|
||||
|
||||
## Documentation
|
||||
|
||||
See **[docs/plugin-eval.md](../../docs/plugin-eval.md)** for the full reference: layers, dimensions, scoring formula, anti-patterns, statistical methods, and project structure.
|
||||
|
||||
Reference in New Issue
Block a user