Adds a GitHub Action that runs the full plugin-eval engine across every
local plugin and emits a structured report for review — not a PR gate.
- scripts/eval_all.py batch-runs EvalEngine.evaluate_plugin across
plugins/*/, writing per-plugin JSON plus summary.md and summary.json
- Report surfaces composite score, 95% CI, badge, confidence label,
anti-pattern flags, and three weakest dimensions per plugin
- Separate "Issues requiring attention" section for plugins scoring
under 60 or with anti-pattern flags raised
- Workflow triggers: workflow_dispatch (choose depth + optional
comma-separated plugin filter) and weekly cron (Mondays 06:00 UTC)
- Depth quick = static only (no credentials); standard/deep require
ANTHROPIC_API_KEY repo secret for LLM judge / Monte Carlo layers
- Posts summary.md to the job summary, uploads full eval-reports/
directory as an artifact for 30 days
Smoke-tested locally: 77 plugins evaluated at quick depth in < 5s,
mean 83.1/100, 2 plugins flagged (incident-response with
DEAD_CROSS_REF, plugin-eval with ORPHAN_REFERENCE + DEAD_CROSS_REF).
Seth Hobson