wshobson-agents

admin/wshobson-agents

Fork 0

mirror of https://github.com/wshobson/agents synced 2026-06-21 14:13:58 +00:00

Commit Graph

Author	SHA1	Message	Date
Seth Hobson	98862b56d8	feat: AGENTS.md canonical context + OpenAI harness-engineering layout (#542 ) * feat: AGENTS.md canonical context + OpenAI harness-engineering layout Promote AGENTS.md to the committed cross-harness context file (per the agents.md convention and OpenAI's harness-engineering blog). Harness- specific files become thin redirects: - AGENTS.md — canonical, committed (~74 lines, table-of-contents) - CLAUDE.md — `@AGENTS.md` import + Claude-specific addenda - GEMINI.md — Gemini-specific setup only - .gemini/settings.json — redirects Gemini CLI's context to read AGENTS.md - ARCHITECTURE.md — new at root, top-level architectural map - gemini-extension.json — bumps version to 1.7.0, sets contextFileName: AGENTS.md - .gitignore — drops the AGENTS.md entry (file is now committed) Harness support verified: - Codex CLI reads AGENTS.md natively (root → cwd walk, 32 KiB cap) - Cursor 2.5+ reads AGENTS.md natively - OpenCode reads AGENTS.md natively (wins over CLAUDE.md if both exist) - Claude Code: `CLAUDE.md` first line is `@AGENTS.md` (Anthropic's documented interop pattern) - Gemini CLI: `.gemini/settings.json` context.fileName redirect (Gemini doesn't support @-imports) Codex adapter no longer generates AGENTS.md — `emit_global` instead validates the committed file fits Codex's 32 KiB cap and the 150-line table-of-contents convention. Tests updated. Clean-output target no longer touches AGENTS.md. ## Auxiliary files updated for multi-harness reality - `.github/ISSUE_TEMPLATE/bug_report.yml` — dropdown for harness + component path; renames "subagent" → "plugin/agent/skill/command" - `.github/ISSUE_TEMPLATE/feature_request.yml` — scope dropdown covers framework / harness / tooling / docs / CI in addition to components - `.github/ISSUE_TEMPLATE/new_subagent.yml` — relabeled "New Component Proposal" with component-type dropdown (plugin/agent/skill/command/ harness adapter) and cross-harness portability field - `.github/ISSUE_TEMPLATE/config.yml` — links to AGENTS.md, authoring guide, per-harness docs; updated Contributing link to root - `.github/CONTRIBUTING.md` — thin pointer to canonical root CONTRIBUTING.md - `.github/PULL_REQUEST_TEMPLATE.md` — new; scope + affected-harness checklists, test-plan checklist, portability-notes section - CONTRIBUTING.md (root) — updated to reference AGENTS.md / ARCHITECTURE.md - gemini-extension.json — version 1.6.0 → 1.7.0, count fixes, redirects to AGENTS.md as contextFileName ## Code-quality CI New `.github/workflows/code-quality.yml` with three jobs: - `python-lint` — `ruff check`, `ruff format --check`, `ty check` on the adapter framework + plugin-eval. yt-design-extractor.py legacy code excluded. - `markdown-lint` — markdownlint-cli2 against README, AGENTS, ARCHITECTURE, CLAUDE, top-level guides, and docs/. Config in `.markdownlint.json`. - `json-lint` — validates every JSON / TOML / YAML in the repo (excluding generated trees). Required ty environment config added to plugin-eval/pyproject.toml so `tools.adapters.` resolves from outside the package. Fixed one ty error in `tools/adapters/base.py:HarnessAdapter.capabilities` (return-type annotation didn't match the `Capability` dataclass returned). Fixed two ruff SIM108 ternary suggestions in codex.py and doc_gardener.py. ruff format applied across all in-scope files (formatting-only diffs). ## Tests + verification - 387 pytest tests pass (1 new test for the AGENTS.md validate-don't-overwrite behavior) - `make validate STRICT=1` clean - `make garden` 0 errors (10 warnings — remaining oversize source skills) - `make smoke-test` clean against locally installed OpenCode/Gemini/Codex/Claude Code - Real-CLI round-trip: `opencode agent list` discovers 193 subagents, `gemini extensions validate .` succeeds, all 191 Codex agent TOMLs parse ## Tag recommendations (separate task — for repo About panel) Top 20 by reach + relevance (from `gh api search/repositories?q=topic:<tag>`): automation mcp ai-agents developer-tools claude-code anthropic agentic-ai agents prompt-engineering cursor multi-agent agent-skills orchestration opencode workflows gemini-cli codex-cli claude-code-skills cursor-rules claude-code-plugins fix(ci): YAML multi-doc + markdownlint scope/rules Two CI failures on PR #542, both fixed: ## JSON/TOML/YAML syntax job (3 false-positive YAML errors) The job's YAML validation used `yaml.safe_load` which only reads the first document in a multi-document YAML stream. Three Kubernetes manifest templates use the standard `---` document separator (valid YAML) and were mis-flagged: plugins/kubernetes-operations/skills/k8s-manifest-generator/assets/configmap-template.yaml plugins/kubernetes-operations/skills/k8s-manifest-generator/assets/service-template.yaml plugins/kubernetes-operations/skills/k8s-security-policies/assets/network-policy-template.yaml Switched to `list(yaml.safe_load_all(...))` so multi-doc YAML is accepted. ## Markdown lint job (lots of pre-existing plugin-README violations) Two changes: 1. Narrow the lint glob — markdownlint now runs against top-level guides (README, AGENTS, ARCHITECTURE, CLAUDE, per-harness setup, CONTRIBUTING) and our authored `docs/` only. Per-plugin READMEs (`plugins//README.md`) are owned by their plugin authors and not lint-gated as part of this framework PR. Lint enforcement for those belongs at the plugin-author layer, not the framework PR layer. 2. Tighten `.markdownlint.json`* — disable two rules that produce noise without catching real defects: - MD040 (fenced-code-language) — terminal output / shell command blocks frequently omit a language by convention - MD060 (table-column-style) — cosmetic table-pipe spacing; doesn't affect rendering Genuine formatting rules kept: MD029 (ol-prefix), MD031 (blanks- around-fences), MD032 (blanks-around-lists), MD056 (table-column- count), MD058 (blanks-around-tables). ## Real defects caught and fixed The narrower scope still caught 5 real issues: - `docs/agent-skills.md:397` — code fence inside an ordered list item needed a blank line before the fence - `docs/authoring.md:90` — bulleted list needed a blank line above - `docs/plugin-eval.md:87` — table needed a blank line above - `OPENCODE.md:39` — table-column-count error caused by literal `\|` inside backticks: ``mode: primary\|subagent\|all`` (3 cells reads as 5) Rewrote as ``mode:` one of `primary` / `subagent` / `all`` ## Verification - `npx markdownlint-cli2 ".md" "docs/.md"` → 0 errors - `yaml.safe_load_all` accepts all multi-doc YAMLs (0 errors) - All other CI jobs already passing (Python ruff/ty, multi-harness generate, CLI smoke test, plugin-eval pytest, tools pytest) * fix: address PR #542 bot feedback - Codex P2 (chatgpt-codex-connector): emit_global now reads AGENTS.md from the repo root (WORKTREE), not output_root. Previously `--output-root <scratch>` produced a false "missing" warning even when AGENTS.md was committed at the real root, breaking --strict generation outside the repo. Added a constructor arg `repo_root` so tests can stage a fake AGENTS.md without touching the committed file, plus a regression test that proves the two paths are decoupled. - CodeRabbit nitpick (code-quality.yml): added workflow-level `permissions: contents: read` and `persist-credentials: false` on every checkout. Skipped the SHA-pinning recommendation — it's a heavier blanket-policy decision and the workflow has no write scope to abuse. - CodeRabbit nitpick (pyproject.toml): consolidated the duplicate `dev` groups by moving `ty` into `[project.optional-dependencies].dev` and removing the now-empty `[dependency-groups]` block. Dropped `--group dev` from `code-quality.yml`'s `uv sync` since `--all-extras` now covers it. - Verified gemini-extension.json counts (82/191/155/102) against the actual source-of-truth: 81 local plugins + 1 external = 82 in marketplace.json, 191 agent .md files, 155 SKILL.md files, 102 command .md files. Counts are correct as-is — CodeRabbit's quick-win was a regex miscount.	2026-05-22 11:16:39 -04:00
Seth Hobson	be57c0b2e3	feat: multi-harness plugin marketplace (Codex, Cursor, OpenCode, Gemini) (#541 ) * feat(adapters): multi-harness framework + harness_portability eval dimension Turn this Claude Code plugin marketplace into a generic agentic-harness marketplace. Adapters under tools/adapters/ emit harness-native artifacts for OpenAI Codex CLI, Cursor, OpenCode, and Gemini CLI from a single Markdown source. Source-of-truth stays under plugins/ — Claude Code is unchanged. Framework (tools/adapters/): - base.py — PluginSource parser, HarnessAdapter ABC, write/mirror helpers (path-traversal guard, UTF-8-safe), inline-list + block-list + block-scalar YAML-ish parser, _utf8_safe_cut, _split_inline_list, _normalize_author - capabilities.py — per-harness capability matrix, TOOL_NAME_MAPS, MODEL_ALIASES, resolve_model() with explicit warnings - codex.py — emits .codex/{skills,agents}/ + AGENTS.md (≤150-line table-of-contents). Fence-aware body splitter, _utf8_safe_cut for multibyte safety, _yaml_scalar with reserved-word + special-char quoting. Skill/command name collision detection (and second-order __cmd fallback). - cursor.py — emits .cursor-plugin/{plugin,marketplace}.json + curated .cursor/rules/.mdc. _validate_mdc_frontmatter handles YAML block scalars (no false positives on colons in description body). _normalize_author handles dict, npm-style strings, and author lists. - opencode.py — transpiles agents to .opencode/agents/<id>.md with mode:subagent + permission: deny-everything-else block (skill/task always allowed as base capabilities — Claude's implicit defaults). - gemini.py — emits native skills/, agents/, and commands/ at extension root (April 2026 spec). Tool-allowlist remapped via TOOL_NAME_MAPS. CLI + tooling: - tools/generate.py — unified `make generate HARNESS=<x> [PLUGIN=<y>]`, with --clean (containment-guarded; case-insensitive on Darwin/Win32), --prune (orphan removal across all per-harness output trees), --strict (warnings fail), per-plugin error aggregation, refuses --clean --plugin (would silently wipe other plugins' artifacts). - tools/validate_generated.py — structural validation across all four harness outputs. Codex 8KB cap → error. _extract_permission_block correctly handles nested permission keys (column-0 only). - tools/doc_gardener.py — recurring drift detection per OpenAI harness- engineering principle. STALE_ARTIFACT (info), DEAD_LINK (error), MARKETPLACE_ORPHAN (error), SKILL_OVER_CODEX_CAP (warning), grouped output sorted by severity. plugin-eval (extends existing framework): - New harness_portability dimension (6% weight, rebalanced from existing static sub-scores). Surfaces non-portable patterns with concrete remediation hints: SKILL_OVER_CODEX_CAP, CLAUDE_TOOL_REFS, CLAUDE_TOOL_PROSE, AGENT_NAME_COLLISION, BARE_MODEL_ALIAS. - _CAMEL_TOOL_PATTERN requires Claude-tool context (no false positives on Rust's `Task` etc.). _TOOL_PROSE_PATTERN case-sensitive on tool names, case-insensitive on the leading article. - Findings do NOT also feed anti_pattern_penalty (no double-counting). Documentation: - Top-level guides: CODEX.md, CURSOR.md, OPENCODE.md (≤150 lines each, table-of-contents pattern per OpenAI harness-engineering post) - docs/harnesses.md — capability matrix, graceful-degradation table, generated output paths - docs/authoring.md — portable-content style guide (tools, models, collision rules, fence-respect) - docs/round-trip-results.md — real-CLI verification recipes (OpenCode discovers 193 subagents, Gemini extensions validate passes, Codex TOMLs all parse) - CONTRIBUTING.md — new file pointing at docs/authoring.md - README.md — rewritten for multi-harness (145 lines, was 460) - CLAUDE.md — trimmed to 60-line table-of-contents - GEMINI.md — trimmed from 1500 to 500 tokens (3× over budget previously) Tests: 181 passing (103 plugin-eval + 78 tools/tests). Real-CLI round-trip verified for OpenCode, Gemini, and Codex (TOML parses). Replaces tools/generate_gemini_commands.py with the unified CLI. refactor(skills): extract detail to references/details.md (~75 skills) Apply Anthropic's canonical SKILL.md progressive-disclosure pattern across the marketplace: SKILL.md body becomes a navigation tier (trigger phrasing + quick start), detailed templates and worked examples move to references/details.md (loaded on demand by the agent). Motivation: OpenAI Codex CLI hard-truncates skills at 8 KB. Before this change, ~90 skills exceeded that cap and would silently break on Codex. The progressive-disclosure pattern is also Anthropic's documented recommendation for token efficiency — Claude Code reads references/ files on demand when the body navigation says to. What's extracted, by pattern: - Pass 1 (## Templates section): 19 skills — full template libraries moved to references/details.md - Pass 2 (## Implementation Patterns / ## Advanced Patterns): 13 skills - Pass 3 (everything between nav-tier and wrap-tier headings): 53 skills - Conservative re-extraction for 8 skills that got over-reduced — kept ~6-7 KB inline (most of the quick-start tier) plus references/ overflow What stays inline (SKILL.md navigation tier): - description: frontmatter (triggering — unchanged for all skills) - ## When to Use This Skill / ## Core Concepts / ## Quick Start - ## Best Practices / ## Troubleshooting / ## See Also wrap-ups - A pointer note ("see references/details.md") so the agent knows where to look for detail What goes to references/details.md (detail tier, on-demand load): - ## Templates (full code template libraries) - ## Implementation Patterns / ## Advanced Patterns (deep examples) - Mid-skill walkthroughs that exceed the inline budget Also in this commit: - plugins/brand-landingpage description trimmed from 958→543 chars (preserves trigger phrasing, drops verbose example-quote list) Net effect: - SKILL_OVER_CODEX_CAP findings: 90 → 10 (88% reduction) - All triggers unchanged — discovery behavior identical across harnesses - 75 new references/details.md files with the extracted content - Same depth of guidance, loaded progressively Remaining 10 oversized skills are complex multi-section docs (e.g. postgresql, code-review-excellence, evaluation-methodology) that need per-skill manual judgment — flagged by `make garden` for future work. * chore: bump all plugin versions (multi-harness release) Patch-bump every local plugin (81) in both .claude-plugin/marketplace.json entries and each plugins/<name>/.claude-plugin/plugin.json. Minor-bump the top-level marketplace metadata.version (1.6.0 → 1.7.0) to signal the multi-harness adapter framework addition. The external git-subdir entry (qa-orchestra) is unaffected — its version is governed by its upstream repo. * fix(opencode): preserve explicit tools:[] + word-boundary subtask match Addresses two Codex review findings on PR #541. ## P1 — `tools: []` silently upgraded to permissive (privilege escalation) Before: `_build_permission_block` returned `{}` for any empty list, which omits the `permission:` block entirely from the emitted agent. An author who explicitly wrote `tools: []` to lock down an advisory-only agent got an UNRESTRICTED agent in OpenCode. Affected agent in this tree: `plugins/arm-cortex-microcontrollers/agents/arm-cortex-expert.md`. Fix: `_build_permission_block` now takes a `has_tools_field` flag so the caller can distinguish "tools: key missing" (Claude default permissive) from "tools: []" (explicit lock-down). The lock-down case emits a deny-everything block that allows ONLY the base capabilities (skill, task) that Claude Code always grants implicitly. Verified against the real arm-cortex-expert agent — now emits read/edit/write/bash/grep/glob/list: deny, task/skill: allow. ## P2 — `"agent" in cmd.body.lower()` false-positives on substrings Before: a command body containing `PerformanceReviewAgent` (class name in a code snippet) or `useragent` triggered `subtask: true`, changing runtime behavior based on incidental text. Fix: switch to a compiled word-boundary regex `\b(agent\|subagent)s?\b` (case-insensitive). Tests confirm the substring `PerformanceReviewAgent` no longer fires, while a real "spawn a subagent" sentence still does. ## Tests 3 new regression tests in tools/tests/test_adapters.py: - `test_explicit_empty_tools_yields_locked_permission_block` (P1) - `test_missing_tools_field_yields_no_permission_block` (P1 boundary) - `test_subtask_inference_word_boundary` (P2) 184 total tests pass (was 181). OpenCode round-trip still discovers all 193 subagents; arm-cortex-expert agent is now properly locked down. * test: behavioral verification + CI gates for multi-harness pipeline Adds three layers of automated verification that pure-Python parser tests miss, plus the CI jobs that turn them into hard gates. Catches the kinds of issues that previously only surfaced when a real user installed the marketplace and tried to use it. ## test_real_world.py — real-source structural tests Runs against the actual `plugins/` tree (not synthetic fixtures). Catches issues that only appear on real content: - every marketplace entry resolves to a plugins/<name>/ dir - every local plugin dir appears in marketplace.json - marketplace.json version == per-plugin plugin.json version (catches drift) - every plugin loads via load_plugin() without error - no plugin name contains `__` (adapter namespace separator) - every agent has name + description; every skill has a trigger phrase (same regex plugin_eval's MISSING_TRIGGER check uses) - no agent name collides with Codex built-ins - every refactored skill (with `references/details.md`) has: - meaningful detail content (>=500 B in details.md) - a pointer to references/ in the SKILL.md body - a navigation-tier heading preserved (When to Use, Overview, etc.) - body >= 600 B (not a stub) - every plugin.json has name + version matching the dir This test pass found and fixed three real defects before commit: - ship-mate/skills/scan: description had no trigger phrase ("Use when…") - reverse-engineering/skills/memory-forensics: nav-tier section lost during extraction - reverse-engineering/skills/binary-analysis-patterns: same All three are now fixed (preserved trigger phrasing, added When-to-Use sections back to the skills my extraction over-trimmed). ## test_round_trip.py — generate→parse→verify CI runs this AFTER `make generate-all`. Catches generation-time regressions: - OpenCode/Codex/Gemini agent counts match source agent count (no skips) - every Codex SKILL.md under 8 KB (the cap that would silently truncate) - every Codex agent TOML has required fields + valid sandbox_mode - every OpenCode agent has mode in {primary,subagent,all} and provider-prefixed model - locked agents (source `tools: []`) emit proper deny-everything permission block with skill/task allow (regression guard for PR-541 P1) - every Gemini @{path} injection resolves to a real source file - every Gemini command TOML has prompt + {{args}} placeholder - every context file (CLAUDE.md, AGENTS.md, GEMINI.md, etc.) within 150-line cap - Cursor marketplace + per-plugin manifests cover all local plugins - .cursor/rules/*.mdc only use the 3 documented frontmatter keys ## test_cli_smoke.py — real-CLI subprocess tests Invokes the actual harness binaries (OpenCode, Gemini, Codex, Claude Code) against the generated artifacts. Catches CLI-level issues pure-Python parsing can't see: schema-loader drift, plugin-discovery bugs, version incompatibilities. - `opencode agent list` — must succeed AND discover every source agent (currently 191 + 2 OpenCode built-ins) - `gemini extensions validate <repo>` — must return success - `codex doctor` — must report healthy install - every Codex agent TOML must parse with stdlib `tomllib` - `claude --version` — sanity check the Claude Code CLI loads - marketplace.json must have owner + metadata.version for Claude Code's loader Per-CLI tests skip gracefully when the binary isn't on PATH, so local devs only exercise what they have installed. CI installs OpenCode + Gemini and turns those skips into hard gates. ## Makefile + CI - `make test` — full pytest suite (plugin-eval + tools/tests/) - `make smoke-test` — generates if needed, then runs real-CLI smoke tests - `.github/workflows/validate.yml` extended with: - `tools-tests` job — runs pytest tools/tests/ - `multi-harness-generate` job — `make generate-all && make validate STRICT=1 && make garden`, uploads generated artifacts on every run - `cli-smoke-test` job — installs OpenCode + Gemini, runs test_cli_smoke.py ## Test counts - Before: 184 tests - After: 386 tests (parameterized real-source tests over all 82 plugins) - All passing locally on OpenCode 1.15.7 + Gemini 0.42.0 + Codex 0.133.0 + Claude Code 2.1.148	2026-05-22 08:18:21 -04:00
Seth Hobson	91fe43e152	docs(plugin-eval): add CLAUDE.md, full evaluation framework docs, and README updates - Create CLAUDE.md with project conventions and PluginEval quick reference - Add docs/plugin-eval.md with complete framework documentation (layers, dimensions, scoring, anti-patterns, statistical methods, project structure) - Add PluginEval section to main README under What's New - Expand plugin-eval/README.md with quick start and command table	2026-03-26 17:40:56 -04:00

Author

SHA1

Message

Date

Seth Hobson

98862b56d8

feat: AGENTS.md canonical context + OpenAI harness-engineering layout (#542 )

* feat: AGENTS.md canonical context + OpenAI harness-engineering layout

Promote AGENTS.md to the committed cross-harness context file (per the
agents.md convention and OpenAI's harness-engineering blog). Harness-
specific files become thin redirects:

- AGENTS.md          — canonical, committed (~74 lines, table-of-contents)
- CLAUDE.md          — `@AGENTS.md` import + Claude-specific addenda
- GEMINI.md          — Gemini-specific setup only
- .gemini/settings.json — redirects Gemini CLI's context to read AGENTS.md
- ARCHITECTURE.md    — new at root, top-level architectural map
- gemini-extension.json — bumps version to 1.7.0, sets contextFileName: AGENTS.md
- .gitignore         — drops the AGENTS.md entry (file is now committed)

Harness support verified:
- Codex CLI reads AGENTS.md natively (root → cwd walk, 32 KiB cap)
- Cursor 2.5+ reads AGENTS.md natively
- OpenCode reads AGENTS.md natively (wins over CLAUDE.md if both exist)
- Claude Code: `CLAUDE.md` first line is `@AGENTS.md` (Anthropic's
  documented interop pattern)
- Gemini CLI: `.gemini/settings.json` context.fileName redirect
  (Gemini doesn't support @-imports)

Codex adapter no longer generates AGENTS.md — `emit_global` instead
validates the committed file fits Codex's 32 KiB cap and the 150-line
table-of-contents convention. Tests updated. Clean-output target no
longer touches AGENTS.md.

## Auxiliary files updated for multi-harness reality

- `.github/ISSUE_TEMPLATE/bug_report.yml` — dropdown for harness +
  component path; renames "subagent" → "plugin/agent/skill/command"
- `.github/ISSUE_TEMPLATE/feature_request.yml` — scope dropdown covers
  framework / harness / tooling / docs / CI in addition to components
- `.github/ISSUE_TEMPLATE/new_subagent.yml` — relabeled "New Component
  Proposal" with component-type dropdown (plugin/agent/skill/command/
  harness adapter) and cross-harness portability field
- `.github/ISSUE_TEMPLATE/config.yml` — links to AGENTS.md, authoring
  guide, per-harness docs; updated Contributing link to root
- `.github/CONTRIBUTING.md` — thin pointer to canonical root CONTRIBUTING.md
- `.github/PULL_REQUEST_TEMPLATE.md` — new; scope + affected-harness
  checklists, test-plan checklist, portability-notes section
- CONTRIBUTING.md (root) — updated to reference AGENTS.md / ARCHITECTURE.md
- gemini-extension.json — version 1.6.0 → 1.7.0, count fixes, redirects
  to AGENTS.md as contextFileName

## Code-quality CI

New `.github/workflows/code-quality.yml` with three jobs:
- `python-lint` — `ruff check`, `ruff format --check`, `ty check` on
  the adapter framework + plugin-eval. yt-design-extractor.py legacy
  code excluded.
- `markdown-lint` — markdownlint-cli2 against README, AGENTS, ARCHITECTURE,
  CLAUDE, top-level guides, and docs/. Config in `.markdownlint.json`.
- `json-lint` — validates every JSON / TOML / YAML in the repo (excluding
  generated trees).

Required ty environment config added to plugin-eval/pyproject.toml so
`tools.adapters.*` resolves from outside the package.

Fixed one ty error in `tools/adapters/base.py:HarnessAdapter.capabilities`
(return-type annotation didn't match the `Capability` dataclass returned).
Fixed two ruff SIM108 ternary suggestions in codex.py and doc_gardener.py.
ruff format applied across all in-scope files (formatting-only diffs).

## Tests + verification

- 387 pytest tests pass (1 new test for the AGENTS.md validate-don't-overwrite behavior)
- `make validate STRICT=1` clean
- `make garden` 0 errors (10 warnings — remaining oversize source skills)
- `make smoke-test` clean against locally installed OpenCode/Gemini/Codex/Claude Code
- Real-CLI round-trip: `opencode agent list` discovers 193 subagents,
  `gemini extensions validate .` succeeds, all 191 Codex agent TOMLs parse

## Tag recommendations (separate task — for repo About panel)

Top 20 by reach + relevance (from `gh api search/repositories?q=topic:<tag>`):
automation mcp ai-agents developer-tools claude-code anthropic agentic-ai
agents prompt-engineering cursor multi-agent agent-skills orchestration
opencode workflows gemini-cli codex-cli claude-code-skills cursor-rules
claude-code-plugins

* fix(ci): YAML multi-doc + markdownlint scope/rules

Two CI failures on PR #542, both fixed:

## JSON/TOML/YAML syntax job (3 false-positive YAML errors)

The job's YAML validation used `yaml.safe_load` which only reads the first
document in a multi-document YAML stream. Three Kubernetes manifest
templates use the standard `---` document separator (valid YAML) and were
mis-flagged:

  plugins/kubernetes-operations/skills/k8s-manifest-generator/assets/configmap-template.yaml
  plugins/kubernetes-operations/skills/k8s-manifest-generator/assets/service-template.yaml
  plugins/kubernetes-operations/skills/k8s-security-policies/assets/network-policy-template.yaml

Switched to `list(yaml.safe_load_all(...))` so multi-doc YAML is accepted.

## Markdown lint job (lots of pre-existing plugin-README violations)

Two changes:

1. **Narrow the lint glob** — markdownlint now runs against top-level
   guides (README, AGENTS, ARCHITECTURE, CLAUDE, per-harness setup,
   CONTRIBUTING) and our authored `docs/` only. Per-plugin READMEs
   (`plugins/*/README.md`) are owned by their plugin authors and not
   lint-gated as part of this framework PR. Lint enforcement for those
   belongs at the plugin-author layer, not the framework PR layer.

2. **Tighten `.markdownlint.json`** — disable two rules that produce
   noise without catching real defects:
   - MD040 (fenced-code-language) — terminal output / shell command
     blocks frequently omit a language by convention
   - MD060 (table-column-style) — cosmetic table-pipe spacing; doesn't
     affect rendering

   Genuine formatting rules kept: MD029 (ol-prefix), MD031 (blanks-
   around-fences), MD032 (blanks-around-lists), MD056 (table-column-
   count), MD058 (blanks-around-tables).

## Real defects caught and fixed

The narrower scope still caught 5 real issues:

- `docs/agent-skills.md:397` — code fence inside an ordered list item
  needed a blank line before the fence
- `docs/authoring.md:90` — bulleted list needed a blank line above
- `docs/plugin-eval.md:87` — table needed a blank line above
- `OPENCODE.md:39` — table-column-count error caused by literal `|`
  inside backticks: ``mode: primary|subagent|all`` (3 cells reads as 5)
  Rewrote as ``mode:` one of `primary` / `subagent` / `all``

## Verification

- `npx markdownlint-cli2 "*.md" "docs/*.md"` → 0 errors
- `yaml.safe_load_all` accepts all multi-doc YAMLs (0 errors)
- All other CI jobs already passing (Python ruff/ty, multi-harness
  generate, CLI smoke test, plugin-eval pytest, tools pytest)

* fix: address PR #542 bot feedback

- Codex P2 (chatgpt-codex-connector): emit_global now reads AGENTS.md from
  the repo root (WORKTREE), not output_root. Previously `--output-root <scratch>`
  produced a false "missing" warning even when AGENTS.md was committed at the
  real root, breaking --strict generation outside the repo. Added a constructor
  arg `repo_root` so tests can stage a fake AGENTS.md without touching the
  committed file, plus a regression test that proves the two paths are decoupled.

- CodeRabbit nitpick (code-quality.yml): added workflow-level `permissions:
  contents: read` and `persist-credentials: false` on every checkout. Skipped
  the SHA-pinning recommendation — it's a heavier blanket-policy decision and
  the workflow has no write scope to abuse.

- CodeRabbit nitpick (pyproject.toml): consolidated the duplicate `dev` groups
  by moving `ty` into `[project.optional-dependencies].dev` and removing the
  now-empty `[dependency-groups]` block. Dropped `--group dev` from
  `code-quality.yml`'s `uv sync` since `--all-extras` now covers it.

- Verified gemini-extension.json counts (82/191/155/102) against the actual
  source-of-truth: 81 local plugins + 1 external = 82 in marketplace.json,
  191 agent .md files, 155 SKILL.md files, 102 command .md files. Counts are
  correct as-is — CodeRabbit's quick-win was a regex miscount.

2026-05-22 11:16:39 -04:00

Seth Hobson

be57c0b2e3

feat: multi-harness plugin marketplace (Codex, Cursor, OpenCode, Gemini) (#541 )

* feat(adapters): multi-harness framework + harness_portability eval dimension

Turn this Claude Code plugin marketplace into a generic agentic-harness
marketplace. Adapters under tools/adapters/ emit harness-native artifacts
for OpenAI Codex CLI, Cursor, OpenCode, and Gemini CLI from a single
Markdown source. Source-of-truth stays under plugins/ — Claude Code is
unchanged.

Framework (tools/adapters/):
- base.py — PluginSource parser, HarnessAdapter ABC, write/mirror helpers
  (path-traversal guard, UTF-8-safe), inline-list + block-list + block-scalar
  YAML-ish parser, _utf8_safe_cut, _split_inline_list, _normalize_author
- capabilities.py — per-harness capability matrix, TOOL_NAME_MAPS,
  MODEL_ALIASES, resolve_model() with explicit warnings
- codex.py — emits .codex/{skills,agents}/ + AGENTS.md (≤150-line
  table-of-contents). Fence-aware body splitter, _utf8_safe_cut for
  multibyte safety, _yaml_scalar with reserved-word + special-char quoting.
  Skill/command name collision detection (and second-order __cmd fallback).
- cursor.py — emits .cursor-plugin/{plugin,marketplace}.json + curated
  .cursor/rules/*.mdc. _validate_mdc_frontmatter handles YAML block scalars
  (no false positives on colons in description body). _normalize_author
  handles dict, npm-style strings, and author lists.
- opencode.py — transpiles agents to .opencode/agents/<id>.md with
  mode:subagent + permission: deny-everything-else block (skill/task always
  allowed as base capabilities — Claude's implicit defaults).
- gemini.py — emits native skills/, agents/, and commands/ at extension
  root (April 2026 spec). Tool-allowlist remapped via TOOL_NAME_MAPS.

CLI + tooling:
- tools/generate.py — unified `make generate HARNESS=<x> [PLUGIN=<y>]`,
  with --clean (containment-guarded; case-insensitive on Darwin/Win32),
  --prune (orphan removal across all per-harness output trees), --strict
  (warnings fail), per-plugin error aggregation, refuses --clean --plugin
  (would silently wipe other plugins' artifacts).
- tools/validate_generated.py — structural validation across all four
  harness outputs. Codex 8KB cap → error. _extract_permission_block
  correctly handles nested permission keys (column-0 only).
- tools/doc_gardener.py — recurring drift detection per OpenAI harness-
  engineering principle. STALE_ARTIFACT (info), DEAD_LINK (error),
  MARKETPLACE_ORPHAN (error), SKILL_OVER_CODEX_CAP (warning), grouped
  output sorted by severity.

plugin-eval (extends existing framework):
- New harness_portability dimension (6% weight, rebalanced from existing
  static sub-scores). Surfaces non-portable patterns with concrete
  remediation hints: SKILL_OVER_CODEX_CAP, CLAUDE_TOOL_REFS,
  CLAUDE_TOOL_PROSE, AGENT_NAME_COLLISION, BARE_MODEL_ALIAS.
- _CAMEL_TOOL_PATTERN requires Claude-tool context (no false positives
  on Rust's `Task` etc.). _TOOL_PROSE_PATTERN case-sensitive on tool
  names, case-insensitive on the leading article.
- Findings do NOT also feed anti_pattern_penalty (no double-counting).

Documentation:
- Top-level guides: CODEX.md, CURSOR.md, OPENCODE.md (≤150 lines each,
  table-of-contents pattern per OpenAI harness-engineering post)
- docs/harnesses.md — capability matrix, graceful-degradation table,
  generated output paths
- docs/authoring.md — portable-content style guide (tools, models,
  collision rules, fence-respect)
- docs/round-trip-results.md — real-CLI verification recipes (OpenCode
  discovers 193 subagents, Gemini extensions validate passes, Codex
  TOMLs all parse)
- CONTRIBUTING.md — new file pointing at docs/authoring.md
- README.md — rewritten for multi-harness (145 lines, was 460)
- CLAUDE.md — trimmed to 60-line table-of-contents
- GEMINI.md — trimmed from 1500 to 500 tokens (3× over budget previously)

Tests: 181 passing (103 plugin-eval + 78 tools/tests). Real-CLI round-trip
verified for OpenCode, Gemini, and Codex (TOML parses).

Replaces tools/generate_gemini_commands.py with the unified CLI.

* refactor(skills): extract detail to references/details.md (~75 skills)

Apply Anthropic's canonical SKILL.md progressive-disclosure pattern across
the marketplace: SKILL.md body becomes a navigation tier (trigger phrasing
+ quick start), detailed templates and worked examples move to
references/details.md (loaded on demand by the agent).

Motivation: OpenAI Codex CLI hard-truncates skills at 8 KB. Before this
change, ~90 skills exceeded that cap and would silently break on Codex.
The progressive-disclosure pattern is also Anthropic's documented
recommendation for token efficiency — Claude Code reads references/ files
on demand when the body navigation says to.

What's extracted, by pattern:
- Pass 1 (## Templates section): 19 skills — full template libraries
  moved to references/details.md
- Pass 2 (## Implementation Patterns / ## Advanced Patterns): 13 skills
- Pass 3 (everything between nav-tier and wrap-tier headings): 53 skills
- Conservative re-extraction for 8 skills that got over-reduced — kept
  ~6-7 KB inline (most of the quick-start tier) plus references/ overflow

What stays inline (SKILL.md navigation tier):
- description: frontmatter (triggering — unchanged for all skills)
- ## When to Use This Skill / ## Core Concepts / ## Quick Start
- ## Best Practices / ## Troubleshooting / ## See Also wrap-ups
- A pointer note ("see references/details.md") so the agent knows where
  to look for detail

What goes to references/details.md (detail tier, on-demand load):
- ## Templates (full code template libraries)
- ## Implementation Patterns / ## Advanced Patterns (deep examples)
- Mid-skill walkthroughs that exceed the inline budget

Also in this commit:
- plugins/brand-landingpage description trimmed from 958→543 chars
  (preserves trigger phrasing, drops verbose example-quote list)

Net effect:
- SKILL_OVER_CODEX_CAP findings: 90 → 10 (88% reduction)
- All triggers unchanged — discovery behavior identical across harnesses
- 75 new references/details.md files with the extracted content
- Same depth of guidance, loaded progressively

Remaining 10 oversized skills are complex multi-section docs (e.g.
postgresql, code-review-excellence, evaluation-methodology) that need
per-skill manual judgment — flagged by `make garden` for future work.

* chore: bump all plugin versions (multi-harness release)

Patch-bump every local plugin (81) in both .claude-plugin/marketplace.json
entries and each plugins/<name>/.claude-plugin/plugin.json. Minor-bump the
top-level marketplace metadata.version (1.6.0 → 1.7.0) to signal the
multi-harness adapter framework addition.

The external git-subdir entry (qa-orchestra) is unaffected — its version
is governed by its upstream repo.

* fix(opencode): preserve explicit tools:[] + word-boundary subtask match

Addresses two Codex review findings on PR #541.

## P1 — `tools: []` silently upgraded to permissive (privilege escalation)

Before: `_build_permission_block` returned `{}` for any empty list, which
omits the `permission:` block entirely from the emitted agent. An author
who explicitly wrote `tools: []` to lock down an advisory-only agent got
an UNRESTRICTED agent in OpenCode. Affected agent in this tree:
`plugins/arm-cortex-microcontrollers/agents/arm-cortex-expert.md`.

Fix: `_build_permission_block` now takes a `has_tools_field` flag so the
caller can distinguish "tools: key missing" (Claude default permissive)
from "tools: []" (explicit lock-down). The lock-down case emits a
deny-everything block that allows ONLY the base capabilities (skill, task)
that Claude Code always grants implicitly. Verified against the real
arm-cortex-expert agent — now emits read/edit/write/bash/grep/glob/list:
deny, task/skill: allow.

## P2 — `"agent" in cmd.body.lower()` false-positives on substrings

Before: a command body containing `PerformanceReviewAgent` (class name
in a code snippet) or `useragent` triggered `subtask: true`, changing
runtime behavior based on incidental text.

Fix: switch to a compiled word-boundary regex `\b(agent|subagent)s?\b`
(case-insensitive). Tests confirm the substring `PerformanceReviewAgent`
no longer fires, while a real "spawn a subagent" sentence still does.

## Tests

3 new regression tests in tools/tests/test_adapters.py:
- `test_explicit_empty_tools_yields_locked_permission_block` (P1)
- `test_missing_tools_field_yields_no_permission_block` (P1 boundary)
- `test_subtask_inference_word_boundary` (P2)

184 total tests pass (was 181). OpenCode round-trip still discovers all
193 subagents; arm-cortex-expert agent is now properly locked down.

* test: behavioral verification + CI gates for multi-harness pipeline

Adds three layers of automated verification that pure-Python parser tests
miss, plus the CI jobs that turn them into hard gates. Catches the kinds
of issues that previously only surfaced when a real user installed the
marketplace and tried to use it.

## test_real_world.py — real-source structural tests

Runs against the actual `plugins/` tree (not synthetic fixtures). Catches
issues that only appear on real content:

- every marketplace entry resolves to a plugins/<name>/ dir
- every local plugin dir appears in marketplace.json
- marketplace.json version == per-plugin plugin.json version (catches drift)
- every plugin loads via load_plugin() without error
- no plugin name contains `__` (adapter namespace separator)
- every agent has name + description; every skill has a trigger phrase
  (same regex plugin_eval's MISSING_TRIGGER check uses)
- no agent name collides with Codex built-ins
- every refactored skill (with `references/details.md`) has:
  - meaningful detail content (>=500 B in details.md)
  - a pointer to references/ in the SKILL.md body
  - a navigation-tier heading preserved (When to Use, Overview, etc.)
  - body >= 600 B (not a stub)
- every plugin.json has name + version matching the dir

This test pass found and fixed three real defects before commit:
- ship-mate/skills/scan: description had no trigger phrase ("Use when…")
- reverse-engineering/skills/memory-forensics: nav-tier section lost
  during extraction
- reverse-engineering/skills/binary-analysis-patterns: same

All three are now fixed (preserved trigger phrasing, added When-to-Use
sections back to the skills my extraction over-trimmed).

## test_round_trip.py — generate→parse→verify

CI runs this AFTER `make generate-all`. Catches generation-time regressions:

- OpenCode/Codex/Gemini agent counts match source agent count (no skips)
- every Codex SKILL.md under 8 KB (the cap that would silently truncate)
- every Codex agent TOML has required fields + valid sandbox_mode
- every OpenCode agent has mode in {primary,subagent,all} and
  provider-prefixed model
- locked agents (source `tools: []`) emit proper deny-everything permission
  block with skill/task allow (regression guard for PR-541 P1)
- every Gemini @{path} injection resolves to a real source file
- every Gemini command TOML has prompt + {{args}} placeholder
- every context file (CLAUDE.md, AGENTS.md, GEMINI.md, etc.) within
  150-line cap
- Cursor marketplace + per-plugin manifests cover all local plugins
- .cursor/rules/*.mdc only use the 3 documented frontmatter keys

## test_cli_smoke.py — real-CLI subprocess tests

Invokes the actual harness binaries (OpenCode, Gemini, Codex, Claude Code)
against the generated artifacts. Catches CLI-level issues pure-Python
parsing can't see: schema-loader drift, plugin-discovery bugs, version
incompatibilities.

- `opencode agent list` — must succeed AND discover every source agent
  (currently 191 + 2 OpenCode built-ins)
- `gemini extensions validate <repo>` — must return success
- `codex doctor` — must report healthy install
- every Codex agent TOML must parse with stdlib `tomllib`
- `claude --version` — sanity check the Claude Code CLI loads
- marketplace.json must have owner + metadata.version for Claude Code's loader

Per-CLI tests skip gracefully when the binary isn't on PATH, so local
devs only exercise what they have installed. CI installs OpenCode +
Gemini and turns those skips into hard gates.

## Makefile + CI

- `make test` — full pytest suite (plugin-eval + tools/tests/)
- `make smoke-test` — generates if needed, then runs real-CLI smoke tests
- `.github/workflows/validate.yml` extended with:
  - `tools-tests` job — runs pytest tools/tests/
  - `multi-harness-generate` job — `make generate-all && make validate
    STRICT=1 && make garden`, uploads generated artifacts on every run
  - `cli-smoke-test` job — installs OpenCode + Gemini, runs test_cli_smoke.py

## Test counts

- Before: 184 tests
- After: 386 tests (parameterized real-source tests over all 82 plugins)
- All passing locally on OpenCode 1.15.7 + Gemini 0.42.0 + Codex 0.133.0
  + Claude Code 2.1.148

2026-05-22 08:18:21 -04:00

Seth Hobson

91fe43e152

docs(plugin-eval): add CLAUDE.md, full evaluation framework docs, and README updates

- Create CLAUDE.md with project conventions and PluginEval quick reference
- Add docs/plugin-eval.md with complete framework documentation (layers,
  dimensions, scoring, anti-patterns, statistical methods, project structure)
- Add PluginEval section to main README under What's New
- Expand plugin-eval/README.md with quick start and command table

2026-03-26 17:40:56 -04:00

3 Commits