mirror of
https://github.com/wshobson/agents
synced 2026-06-21 14:13:58 +00:00
Seth Hobson
2d3f6a8527
* chore: move toolchain to uv-native (no pip, no requirements.txt)
The repo previously mixed uv (for plugin-eval) with pip + requirements.txt
(for yt-design-extractor) and raw `python3` invocations in the Makefile and
CI workflows. This makes the toolchain uniformly uv-managed.
## yt-design-extractor
Moved `tools/yt-design-extractor.py` into `tools/yt-design-extractor/` with its
own `pyproject.toml` + `uv.lock`. EasyOCR (and its ~2 GB torch dependency)
becomes an optional extra (`uv sync --extra easyocr`). Deleted
`tools/requirements.txt`.
## Makefile
All targets now route through uv:
- `make install` / `make install-easyocr` / `make deps` / `make check` / `make run*`:
`cd tools/yt-design-extractor && uv run/uv sync`
- `make generate` / `make validate` / `make garden` / `make clean-generated`:
`uv run --project plugins/plugin-eval python tools/...` (reuses plugin-eval's
venv — it already has pyyaml and `extra-paths = ["../.."]` for tools/adapters)
- `make test` / `make smoke-test`: `uv run --project plugins/plugin-eval pytest`
## CI
- `.github/workflows/validate.yml`:
- `multi-harness-generate` swapped from `actions/setup-python@v5` to
`astral-sh/setup-uv@v5` + `uv sync` of plugin-eval before `make generate-all`
- Workflow-level `permissions: contents: read` + `persist-credentials: false`
on every checkout (carrying the security hardening forward consistently)
- `.github/workflows/code-quality.yml`:
- `json-lint` job swapped from `setup-python` to `setup-uv`
- YAML validator now uses `uv run --with pyyaml python` (no pre-step install)
- JSON/TOML validators use `uv run python` (stdlib `json.tool`, `tomllib`)
- Dropped the standalone `pip install pyyaml --quiet` line
## Inline hints
- `plugins/plugin-eval/src/plugin_eval/layers/judge.py`: error message
recommends `uv sync --extra llm` instead of `pip install plugin-eval[llm]`
- `tools/yt-design-extractor/yt-design-extractor.py`: usage docstring and
install hints now reference `make install` / `make install-easyocr` / `uv run`
The pip refs inside plugin-authored commands (deps-audit.md, doc-generate.md,
error-trace.md) describe scanning **user** Python projects — those legitimately
use pip and are out of scope.
Local gates green: make test (385 pass), make garden (0 errors), make validate
clean, ruff + format + ty all pass, markdownlint clean.
* chore: address PR #543 review feedback
- checkmake (Makefile): consolidate the multi-line `.PHONY:` declaration
onto a single line. The previous backslash-continued form was readable
but checkmake couldn't parse it, falsely flagging `validate` and
`clean-generated` as missing from .PHONY. They were already declared —
just invisible to the linter.
Other CodeRabbit feedback declined:
- pyproject.toml dep version constraints (CodeRabbit self-tagged "Low value"):
uv.lock pins exact versions for reproducibility; upper bounds on yt-dlp,
Pillow, etc. would invite stale-pin churn without changing the locked
installation. Tracking upstream aggressively is the right default for a
utility tool.
- SHA-pinning for GitHub Actions (CodeRabbit "Major" but defensible):
Workflow has no write scope (permissions: contents: read on both files),
no secrets are exposed, and the actions involved are first-party Anthropic
(astral-sh, actions/*, DavidAnson). Same policy decision as PR #542 —
blanket SHA pinning is a bigger commitment than this PR's scope warrants.
* chore: pin all GitHub Actions to commit SHAs
Addresses CodeRabbit's Major finding (zizmor `unpinned-uses`). Reverses the
earlier policy decision now that the workflow scope has grown — pins both
workflows consistently in one pass:
- actions/checkout v4 → 34e114876b0b11c390a56381ad16ebd13914f8d5
- actions/setup-node v4 → 49933ea5288caeca8642d1e84afbd3f7d6820020
- actions/upload-artifact v4 → ea165f8d65b6e75b540449e92b4886f43607fa02
- astral-sh/setup-uv v5 → e58605a9b6da7c637471fab8847a5e5a6b8df081
- DavidAnson/markdownlint-cli2-action v18 → eb5ca3ab411449c66620fe7f1b3c9e10547144b0
- oven-sh/setup-bun v2 → 0c5077e51419868618aeaa5fe8019c62421857d6
Each pin keeps the version tag in a trailing comment for human readability
and dependabot/renovate compatibility.
Both workflows YAML-validated locally; functionality unchanged.
plugin-eval
Three-layer quality evaluation framework for Claude Code plugins.
Quick Start
cd plugins/plugin-eval
uv sync
# Evaluate a skill (static only, instant)
uv run plugin-eval score path/to/skill --depth quick
# Evaluate with LLM judge (~30s)
uv run plugin-eval score path/to/skill --depth standard
# Full certification (all layers, ~5 min)
uv run plugin-eval certify path/to/skill
Layers
- Static Analysis — Structural checks, anti-pattern detection. Instant, free.
- LLM Judge — Semantic evaluation (triggering, orchestration, output, scope). ~30s, 4 calls.
- Monte Carlo — Statistical reliability via 50–100 simulated runs. ~2–5 min.
Commands
| CLI | Claude Code | Description |
|---|---|---|
plugin-eval score |
/eval |
Score a plugin or skill |
plugin-eval certify |
/certify |
Full certification with badge |
plugin-eval compare |
/compare |
Head-to-head comparison |
plugin-eval init |
— | Build corpus for Elo ranking |
Documentation
See docs/plugin-eval.md for the full reference: layers, dimensions, scoring formula, anti-patterns, statistical methods, and project structure.