3 Commits

Author SHA1 Message Date
jon 5162141430 fix(plugin-eval): exempt slash-only and path-triggered skills from MISSING_TRIGGER (#531)
* fix(plugin-eval): exempt slash-only and path-triggered skills from MISSING_TRIGGER

Skills can be invoked through three mechanisms in Claude Code:

1. Model-driven auto-invocation based on the description (the default,
   and what MISSING_TRIGGER is designed to gate).
2. User-driven slash invocation only — opt-out via
   `disable-model-invocation: true` in the SKILL.md frontmatter.
3. Path-triggered auto-load — opt-in via the `paths:` frontmatter glob,
   where the skill is loaded when the model opens a matching file.

The current `MISSING_TRIGGER` check runs against every skill regardless
of invocation mechanism. Skills in (2) cannot be auto-invoked at all, so
a trigger phrase in the description is irrelevant. Skills in (3) are
triggered by file paths, not description matching, so the description
serves as documentation rather than as a discovery surface.

Add a `_skill_uses_description_trigger(skill)` predicate that consults
`skill.frontmatter` and short-circuits the MISSING_TRIGGER check for
the two non-description invocation paths. Behaviour for ordinary
model-invocable skills is unchanged.

Tests:
- `test_disable_model_invocation_exempts_skill` — slash-only, no trigger,
  no flag.
- `test_paths_auto_load_exempts_skill` — path-triggered, no trigger,
  no flag.
- `test_disable_model_invocation_false_still_checks_trigger` — explicit
  `false` does not exempt.
- `test_empty_paths_value_still_checks_trigger` — empty string is not
  a valid auto-load configuration.

Full plugin-eval suite (73 tests) passes.

* fix: address ruff SIM103/SIM102 and guard paths check against non-string values

Simplify conditional returns (SIM103), collapse nested ifs (SIM102),
and use isinstance(paths, str) to prevent paths: [] from incorrectly
exempting a skill from MISSING_TRIGGER.

---------

Co-authored-by: Seth Hobson <wshobson@gmail.com>
2026-05-25 21:14:51 -04:00
jon 112197c6bf fix(plugin-eval): broaden MISSING_TRIGGER pattern to match canonical phrasings (#530)
The static-layer trigger heuristic was a literal substring match on
"Use when …" / "Use this skill when …" / "Use proactively" / "Trigger
when …". Three real-world phrasings were missed:

1. Third-person canonical form recommended by Anthropic's own plugin-dev
   skill-development skill: "This skill should be used when …" (contains
   "used when", not "use when"). Running the previous regex on plugin-dev
   itself produces 7 false-positive MISSING_TRIGGER flags.

2. Prepositional temporal triggers: "Use after …", "Use before …",
   "Use immediately before …", "Use whenever …". Common in self-audit
   and hook-adjacent skills (e.g. functional-emotions: 6 flags).

3. Path-triggered self-documenting skills: "Auto-loads when …".

Extract the pattern to a module-level `_TRIGGER_PATTERN` and share it
between the anti-pattern detector and `_description_pushiness` so the
two sites stay in sync. Expand it to match `(should be )?use(d)?
(this skill )?(immediately )?(when|after|before|whenever)` plus
`auto-loads when` and `trigger when`.

Verification:
- plugin-dev (Anthropic's own): 33.82 → 55.18, MISSING_TRIGGER cleared
- functional-emotions: 39.04 → 59.68, all 6 false positives cleared
- 22 tests pass, including 11 new positive-form parametrised cases
  and 2 regression tests against canonical Anthropic phrasings.
2026-05-14 09:04:34 -04:00
Seth Hobson 882f344786 feat(plugin-eval): implement Layer 1 static analysis with anti-pattern detection 2026-03-25 17:39:58 -04:00