The static-layer trigger heuristic was a literal substring match on
"Use when …" / "Use this skill when …" / "Use proactively" / "Trigger
when …". Three real-world phrasings were missed:
1. Third-person canonical form recommended by Anthropic's own plugin-dev
skill-development skill: "This skill should be used when …" (contains
"used when", not "use when"). Running the previous regex on plugin-dev
itself produces 7 false-positive MISSING_TRIGGER flags.
2. Prepositional temporal triggers: "Use after …", "Use before …",
"Use immediately before …", "Use whenever …". Common in self-audit
and hook-adjacent skills (e.g. functional-emotions: 6 flags).
3. Path-triggered self-documenting skills: "Auto-loads when …".
Extract the pattern to a module-level `_TRIGGER_PATTERN` and share it
between the anti-pattern detector and `_description_pushiness` so the
two sites stay in sync. Expand it to match `(should be )?use(d)?
(this skill )?(immediately )?(when|after|before|whenever)` plus
`auto-loads when` and `trigger when`.
Verification:
- plugin-dev (Anthropic's own): 33.82 → 55.18, MISSING_TRIGGER cleared
- functional-emotions: 39.04 → 59.68, all 6 false positives cleared
- 22 tests pass, including 11 new positive-form parametrised cases
and 2 regression tests against canonical Anthropic phrasings.
`plugin-eval certify <plugin-dir>` advertises a deep, three-layer
evaluation (static + judge + Monte Carlo) but `EvalEngine.evaluate_plugin`
only runs the static layer regardless of requested depth, since judge
and Monte Carlo are per-skill primitives. The docstring on that method
acknowledges this, but nothing surfaces it to the user:
- The CLI emits no warning.
- The markdown report prints `**Depth:** deep` even though only the
static layer ran.
- The user's only signal is a footnote ("No model usage") near the
bottom of the report and the `Confidence: Estimated` row — both
easy to miss when the requested depth said otherwise.
This change makes the downgrade impossible to miss without changing
the underlying eval behaviour (per-skill judge aggregation is a larger
feature, not a bug fix):
1. **CLI warning to stderr.** `_run_score` detects plugin-target runs
at non-quick depth and prints a yellow `warning:` line to stderr
naming the skipped layers and the workaround (run on a single skill
to get the deeper layers).
2. **In-band markdown callout.** `Reporter` now derives the *effective*
depth from the set of layers actually present in the result. When it
differs from the requested depth, the report header reads
`Depth: deep (requested) → quick (effective)` and a `> Note:` block
above the score table explains why and how to get the deeper layers.
3. **Effective-depth helper.** `_effective_depth(result)` maps the set
of layer names (`static` / `judge` / `monte_carlo`) back to a `Depth`
value, so the reporter never has to trust `result.config.depth` when
describing what actually ran.
Tests:
- `TestDepthDowngradeWarning` (4 tests): asserts the helper, the
"no warning when honored" path, and the warning content for both
deep and standard requests.
- `TestCLI` (2 new tests): asserts the stderr warning is emitted on
plugin-level certify and is *not* emitted at quick depth.
Full plugin-eval suite (75 tests) passes.
Adds MonteCarloAnalyzer with SimResult/MonteCarloConfig dataclasses, run_simulation
async helper, and _compute_statistics using Wilson score CI, bootstrap CI,
Clopper-Pearson CI, and coefficient of variation. Wires MC layer into evaluate_skill
for Depth.DEEP and Depth.THOROUGH runs (50 and 100 runs respectively).
Adds EvalEngine that coordinates static analysis, blends layer scores
across dimensions with renormalized weights, and produces PluginEvalResult
with composite score, badge, and per-dimension grades. Layer 2/3 stubs
ready for Tasks 8/9.
jon