mirror of
https://github.com/trailofbits/skills
synced 2026-06-21 14:12:00 +00:00
trailmark: add review gate skill
This commit is contained in:
@@ -225,8 +225,8 @@
|
||||
},
|
||||
{
|
||||
"name": "trailmark",
|
||||
"version": "0.9.0",
|
||||
"description": "Builds multi-language source and binary code graphs for security analysis: call graphs, attack surface mapping, blast radius, taint propagation, complexity hotspots, entry point enumeration, proxy/unresolved-call tracking, type/reference analysis, and structural diffs. Generates Mermaid diagrams, runs graph-informed mutation testing triage (genotoxic), generates mutation-driven test vectors (vector-forge), extracts crypto protocol message flows, converts Mermaid diagrams to ProVerif models, projects SARIF/weAudit/binary findings onto code graphs, and triages single findings with graph evidence. Use when analyzing call paths, mapping attack surface, visualizing code architecture, triaging survived mutants, generating cryptographic test vectors, diagramming crypto protocols, formally verifying protocols, augmenting audits with static analysis findings, or deciding whether one candidate issue is reachable.",
|
||||
"version": "0.10.0",
|
||||
"description": "Builds multi-language source and binary code graphs for security analysis: call graphs, attack surface mapping, blast radius, taint propagation, complexity hotspots, entry point enumeration, proxy/unresolved-call tracking, type/reference analysis, and structural diffs. Generates Mermaid diagrams, runs graph-informed mutation testing triage (genotoxic), generates mutation-driven test vectors (vector-forge), extracts crypto protocol message flows, converts Mermaid diagrams to ProVerif models, projects SARIF/weAudit/binary findings onto code graphs, triages single findings with graph evidence, and gates branch diffs for structural review regressions. Use when analyzing call paths, mapping attack surface, visualizing code architecture, triaging survived mutants, generating cryptographic test vectors, diagramming crypto protocols, formally verifying protocols, augmenting audits with static analysis findings, deciding whether one candidate issue is reachable, or reviewing graph-level PR risk.",
|
||||
"author": {
|
||||
"name": "Scott Arciszewski",
|
||||
"url": "https://github.com/tob-scott-a"
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"name": "trailmark",
|
||||
"version": "0.9.0",
|
||||
"description": "Builds multi-language source and binary code graphs for security analysis: call graphs, attack surface mapping, blast radius, taint propagation, complexity hotspots, entry point enumeration, proxy/unresolved-call tracking, type/reference analysis, and structural diffs. Generates Mermaid diagrams, runs graph-informed mutation testing triage (genotoxic), generates mutation-driven test vectors (vector-forge), extracts crypto protocol message flows, converts Mermaid diagrams to ProVerif models, projects SARIF/weAudit/binary findings onto code graphs, and triages single findings with graph evidence. Use when analyzing call paths, mapping attack surface, visualizing code architecture, triaging survived mutants, generating cryptographic test vectors, diagramming crypto protocols, formally verifying protocols, augmenting audits with static analysis findings, or deciding whether one candidate issue is reachable.",
|
||||
"version": "0.10.0",
|
||||
"description": "Builds multi-language source and binary code graphs for security analysis: call graphs, attack surface mapping, blast radius, taint propagation, complexity hotspots, entry point enumeration, proxy/unresolved-call tracking, type/reference analysis, and structural diffs. Generates Mermaid diagrams, runs graph-informed mutation testing triage (genotoxic), generates mutation-driven test vectors (vector-forge), extracts crypto protocol message flows, converts Mermaid diagrams to ProVerif models, projects SARIF/weAudit/binary findings onto code graphs, triages single findings with graph evidence, and gates branch diffs for structural review regressions. Use when analyzing call paths, mapping attack surface, visualizing code architecture, triaging survived mutants, generating cryptographic test vectors, diagramming crypto protocols, formally verifying protocols, augmenting audits with static analysis findings, deciding whether one candidate issue is reachable, or reviewing graph-level PR risk.",
|
||||
"author": {
|
||||
"name": "Scott Arciszewski",
|
||||
"url": "https://github.com/tob-scott-a"
|
||||
|
||||
@@ -47,6 +47,7 @@ uv pip install trailmark
|
||||
| `genotoxic` | Triage mutation testing results using graph analysis — classify survived mutants as false positives, missing tests, or fuzzing targets |
|
||||
| `vector-forge` | Mutation-driven test vector generation — find coverage gaps via mutation testing, then generate Wycheproof-style vectors that close them |
|
||||
| `graph-evolution` | Compare code graphs at two snapshots to surface security-relevant structural changes text diffs miss |
|
||||
| `trailmark-review-gate` | Apply PASS/WARN/FAIL/UNKNOWN structural gate rules to branch, PR, fix, or release diffs |
|
||||
| `mermaid-to-proverif` | Convert Mermaid sequence diagrams into ProVerif formal verification models |
|
||||
| `audit-augmentation` | Project SARIF, weAudit, and v0.4 binary-analysis graph findings onto code graphs as annotations and subgraphs |
|
||||
| `trailmark-finding-triage` | Triage one finding, SARIF result, weAudit annotation, suspicious function, or report excerpt with reachability, taint, privilege-boundary, and blast-radius evidence |
|
||||
@@ -71,6 +72,7 @@ trailmark/
|
||||
│ └── references/
|
||||
├── graph-evolution/ # Structural diff
|
||||
│ └── scripts/graph_diff.py
|
||||
├── trailmark-review-gate/ # Structural review gates
|
||||
├── mermaid-to-proverif/ # Sequence diagram → ProVerif
|
||||
│ └── examples/
|
||||
├── audit-augmentation/ # SARIF/weAudit integration
|
||||
|
||||
@@ -301,6 +301,12 @@ line-level code review. The two are complementary — graph-evolution
|
||||
finds attack paths that text diffs miss, while differential-review
|
||||
provides git blame context and micro-adversarial analysis.
|
||||
|
||||
**trailmark-review-gate skill:**
|
||||
Use trailmark-review-gate after graph-evolution when a branch, pull request,
|
||||
fix commit, or release diff needs a PASS/WARN/FAIL/UNKNOWN structural review
|
||||
packet. The gate applies deterministic review rules to graph-evolution output;
|
||||
it does not replace human review.
|
||||
|
||||
**genotoxic skill:**
|
||||
If graph-evolution reveals new high-CC tainted nodes, feed them to
|
||||
genotoxic for mutation testing triage.
|
||||
|
||||
@@ -0,0 +1,112 @@
|
||||
---
|
||||
name: trailmark-review-gate
|
||||
description: "Runs a Trailmark structural review gate over a branch, pull request, fix commit, release diff, or git ref range to detect new entrypoints, new tainted paths, removed validation or authorization calls, privilege-boundary drift, blast-radius growth, complexity growth, and newly reachable sensitive sinks. Use when reviewing a PR, branch, remediation commit, or release diff where graph-level security regressions should be checked before merge."
|
||||
allowed-tools:
|
||||
- Bash
|
||||
- Read
|
||||
- Grep
|
||||
- Glob
|
||||
- Write
|
||||
---
|
||||
|
||||
# Trailmark Review Gate
|
||||
|
||||
Apply deterministic security gate rules to Trailmark structural diff evidence.
|
||||
This skill does not replace line-level review. It produces a compact structural
|
||||
packet reviewers can cite while they inspect the code.
|
||||
|
||||
## When to Use
|
||||
|
||||
- Reviewing a branch, pull request, release diff, or fix commit
|
||||
- Checking whether a change expands attack surface
|
||||
- Looking for removed validation or authorization on reachable paths
|
||||
- Comparing before/after taint, privilege-boundary, blast-radius, or
|
||||
complexity signals
|
||||
- Producing graph evidence for a differential review
|
||||
|
||||
## When NOT to Use
|
||||
|
||||
- Single-snapshot analysis. Use `trailmark` or `trailmark-structural`.
|
||||
- Text-diff review only. Use `differential-review`.
|
||||
- Full vulnerability discovery. Use an audit or bug-finding workflow.
|
||||
- One static finding. Use `trailmark-finding-triage`.
|
||||
- Tooling is unavailable and the user wants manual review only.
|
||||
|
||||
## Rationalizations to Reject
|
||||
|
||||
| Rationalization | Why It Is Wrong | Required Action |
|
||||
|---|---|---|
|
||||
| "The line diff is small, so no graph gate is needed" | Small changes can create new call paths | Compare before/after graphs |
|
||||
| "Graph gate passed, so the PR is secure" | The gate only checks structural regressions | Still perform line-level review |
|
||||
| "Trailmark failed, so pass the gate" | Tool failure is unknown risk, not success | Emit `UNKNOWN` |
|
||||
| "Tests pass, so removed validation is fine" | Tests may miss affected entrypoint paths | Review the removed path manually |
|
||||
| "Only new code matters" | Removed auth, validation, and callers can be higher risk than additions | Review removals and path changes |
|
||||
|
||||
## Workflow
|
||||
|
||||
```
|
||||
Review Gate Progress:
|
||||
- [ ] Step 1: Resolve before/after inputs
|
||||
- [ ] Step 2: Build graph-evolution evidence
|
||||
- [ ] Step 3: Normalize structural changes
|
||||
- [ ] Step 4: Apply gate rules
|
||||
- [ ] Step 5: Emit review packet and actions
|
||||
```
|
||||
|
||||
### Step 1: Resolve Inputs
|
||||
|
||||
Accept two refs, a branch name, a commit range, or before/after directories.
|
||||
Do not check out branches unnecessarily. Prefer `git diff`, `git show`, and
|
||||
git worktrees, following the `graph-evolution` snapshot workflow.
|
||||
|
||||
### Step 2: Build Graph Evidence
|
||||
|
||||
Run `graph-evolution` or equivalent Trailmark before/after graph analysis.
|
||||
Both snapshots must run `engine.preanalysis()` so taint, privilege-boundary,
|
||||
blast-radius, complexity, and entrypoint signals are available.
|
||||
|
||||
Record Trailmark version and any feature probes. If graph construction fails,
|
||||
emit `UNKNOWN`.
|
||||
|
||||
### Step 3: Normalize Changes
|
||||
|
||||
Normalize evidence into:
|
||||
|
||||
- added, removed, and modified nodes
|
||||
- added and removed edges
|
||||
- entrypoint set changes
|
||||
- taint membership changes
|
||||
- privilege-boundary membership changes
|
||||
- blast-radius changes
|
||||
- complexity changes
|
||||
- newly reachable sensitive sinks
|
||||
- unresolved, proxy, or dynamic edge changes
|
||||
|
||||
### Step 4: Apply Gate Rules
|
||||
|
||||
Apply the rules in [references/gate-rules.md](references/gate-rules.md).
|
||||
Gate verdicts are:
|
||||
|
||||
| Verdict | Meaning |
|
||||
|---|---|
|
||||
| `FAIL` | A high-risk structural regression needs review before acceptance |
|
||||
| `WARN` | A meaningful graph change needs reviewer attention |
|
||||
| `PASS` | No configured structural gate fired |
|
||||
| `UNKNOWN` | Trailmark failed or evidence is too incomplete |
|
||||
|
||||
### Step 5: Emit Packet
|
||||
|
||||
Write the packet using
|
||||
[references/output-format.md](references/output-format.md), then hand it to
|
||||
the branch reviewer. Use
|
||||
[references/review-integration.md](references/review-integration.md) when
|
||||
combining this packet with `differential-review` or another PR review process.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Never mutate the user's working branch while comparing refs.
|
||||
- Never report `PASS` when Trailmark failed.
|
||||
- Separate graph evidence from manual security judgment.
|
||||
- Include exact changed nodes or paths for every `FAIL` and `WARN`.
|
||||
- Include limitations when parser, proxy, unresolved-call, or dynamic-dispatch
|
||||
uncertainty affects the verdict.
|
||||
@@ -0,0 +1,4 @@
|
||||
interface:
|
||||
icon_small: "assets/trail-of-bits-mark.svg"
|
||||
icon_large: "assets/trail-of-bits-mark.svg"
|
||||
brand_color: "#D83A34"
|
||||
@@ -0,0 +1 @@
|
||||
<svg xmlns="http://www.w3.org/2000/svg" width="94" height="56" fill="none" viewBox="0 0 94 56"><path fill="#F0F4F7" d="m34.04 54.662-7.61-4.147L24.593 56l9.433-1.335c-.029 0-.043 0 .014-.003"/><path fill="#F0F4F7" d="m34.039 54.662-.014.003c.035 0 .096-.003.014-.003m26.191-2.67 6.124-1.804 2.301-7.26-5.655.387zM74.805 5.478l-4.68-3.035-2.62 8.332 5.15 1.548zM43.224 3.532s3.172.973 4.423 1.328l4.508 1.335L52.234 0l-7.928 1.576zm-31.473 23.14 5.566.014 1.982-6.216-5.06-1.342c-.538 1.708-1.94 5.837-2.488 7.544M1.394 20.896l4.164 4.338 2.398-7.696-5.11-1.357zm88.205 24.841c-.086-2.18-.692-2.894-1.978-4.232l-6.71.447c1.871 1.175 3.018 2.63 3.255 4.583.261 2.145-2.068 4.623-4.322 4.623-1.258 0-1.885-.987-1.885-2.12.035-.845.333-1.942.777-2.673h-5.691c-.444 1.136-.813 2.418-.813 3.625 0 4.136 3.659 5.197 7.131 5.197 3.62 0 6.616-.696 8.501-4.03.85-1.505 1.806-3.663 1.735-5.42M18.804.56 1.362.576 0 4.86l6.394-.007-3.161 9.962 5.114 1.356 3.551-11.322 5.544-.007z"/><path fill="#F0F4F7" d="M20.707 15.898c.628-.04 1.258-.04 1.886-.04 1.035-.003 2.587.072 2.587 1.499.004.987-.366 2.233-.66 3.184-.551 1.942-1.214 3.88-1.325 5.858l5.727-.007c-.151-3.185 1.842-5.968 1.838-9.117 0-2.123-1.627-2.964-3.512-3.294l.552-.075c4.103-.554 6.39-3.738 6.386-7.729-.004-4.423-3.738-5.666-7.544-5.662l-6.576.007c-1.87 5.751-3.645 11.534-5.462 17.31l5.057 1.339zM24.245 4.58l1.849-.004c1.369 0 2.734.327 2.738 1.939.003 1.977-1.437 5.31-3.803 5.31l-3.031.004zm11.959 21.883 2.949-5.531 7.06-.008-.215 5.564 5.544-.004.441-18.94-4.763-1.43a89 89 0 0 0-.677 10.71h-.036l-5.322.004c1.914-3.586 3.749-7.2 5.4-10.906L42.77 4.775 30.437 26.466zm39.249-.036 1.402-4.214-7.43.01 2.753-8.612-5.15-1.548-4.584 14.375zm-34.411 1.658h-7.28l-6.834 21.18 8.698 4.694c5.208-.196 8.856-4.012 8.856-9.252 0-2.013-1.258-3.586-3.215-4.136 3.846-.877 6.319-3.33 6.319-7.356-.004-3.923-3.072-5.13-6.544-5.13m-2.993 20.318c-1.036 1.537-2.143 1.647-3.881 1.647h-2.254l2.476-7.611c1.81.07 5.024-.366 5.024 2.268 0 1.168-.698 2.744-1.365 3.696m-.444-9.667h-2.072l2.18-6.7c1.626.075 4.694-.436 4.694 2.053.04 2.928-1.846 4.647-4.802 4.647M58.67 9.582l-5.522 18.29-4.878 14.836 5.856-.447 4.23-13.2h.006l5.713-17.878c-.796-.22-3.964-1.228-5.404-1.601m2.738 18.542-1.37 4.278h6.398l-2.993 9.525 5.584-.38 2.913-9.145h5.541l1.37-4.278zm25.351-.259c-5.726 0-9.127 3.951-9.127 9.444.007.798.727 2.765 2.2 3.422l6.888-.462c-1.172-.98-3.243-3.29-3.243-4.519 0-1.686.95-3.891 2.91-3.891 1.33 0 2.143.366 2.143 1.793 0 .916-.444 1.757-.702 2.638h5.322c.333-.77.849-2.528.849-3.334.004-3.994-3.913-5.09-7.24-5.09m-63.15.372c-2.605 0-2.978 1.906-3.623 3.93-.215.728-.623 1.615-.623 2.379 0 1.47 1.315 1.75 2.537 1.75 1.494.01 2.29-.487 2.82-1.878.362-.952 1.305-3.568 1.305-4.488 0-1.303-1.326-1.693-2.416-1.693m.728 1.87c0 .328-.373 1.392-.498 1.765l-.577 1.782c-.226.675-.498 1.449-1.359 1.449-.487 0-.802-.28-.802-.785 0-.653.509-1.864.724-2.538.215-.671.498-2.01 1.247-2.269.168-.056.351-.078.534-.078.34 0 .749.302.749.664zm5.902 2.674.394-1.292H28.37l.613-1.928h2.38l.42-1.292h-4.126l-2.48 7.924h1.735l1.075-3.412z"/></svg>
|
||||
|
After Width: | Height: | Size: 3.0 KiB |
@@ -0,0 +1,53 @@
|
||||
# Gate Rules
|
||||
|
||||
Start with deterministic, conservative rules. A triggered rule creates a
|
||||
review obligation; it does not prove a vulnerability.
|
||||
|
||||
| Rule | Verdict | Why it matters |
|
||||
|---|---|---|
|
||||
| New untrusted entrypoint | `FAIL` | Expands external attack surface |
|
||||
| New path from untrusted entrypoint to sensitive sink | `FAIL` | Creates a candidate exploit path |
|
||||
| Removed auth, validation, or sanitization call on reachable path | `FAIL` | Common regression in fixes and feature PRs |
|
||||
| Newly tainted privilege-boundary node | `FAIL` | Trust transition now handles untrusted data |
|
||||
| Blast radius growth above threshold | `WARN` | A bug may now affect more code |
|
||||
| Complexity growth on tainted or boundary node | `WARN` | Risky logic became harder to review |
|
||||
| New unresolved, proxy, or dynamic call on reachable path | `WARN` | Graph uncertainty increased in a risky area |
|
||||
| Dead security function removed | `WARN` | May be cleanup or accidental security removal |
|
||||
|
||||
## Default Thresholds
|
||||
|
||||
Use these defaults unless the repository has stricter local rules:
|
||||
|
||||
| Signal | Default threshold |
|
||||
|---|---|
|
||||
| Blast radius growth | `+5` downstream reachable nodes or `+25%`, whichever is larger |
|
||||
| Complexity growth | cyclomatic complexity `+3` on tainted, boundary, or entrypoint-reachable node |
|
||||
| Sensitive sink path | any new path from untrusted entrypoint to sink |
|
||||
| Unresolved/proxy growth | any new unresolved/proxy edge on an entrypoint-reachable path |
|
||||
|
||||
Thresholds are intentionally conservative. They reduce noise while still
|
||||
catching structural changes that line diffs often understate.
|
||||
|
||||
## Sensitive Sink Categories
|
||||
|
||||
Flag new reachable paths to:
|
||||
|
||||
- value transfer
|
||||
- authorization or role decisions
|
||||
- persistence or state writes
|
||||
- parsing or deserialization
|
||||
- cryptographic keys, sessions, or signatures
|
||||
- external process, network, or file operations
|
||||
- upgrade, plugin, hook, or dynamic dispatch mechanisms
|
||||
|
||||
## Rule Precedence
|
||||
|
||||
Use the most severe triggered verdict:
|
||||
|
||||
1. `UNKNOWN` if Trailmark cannot produce adequate evidence
|
||||
2. `FAIL` if any fail rule triggers
|
||||
3. `WARN` if any warn rule triggers
|
||||
4. `PASS` only if evidence is adequate and no rule triggers
|
||||
|
||||
If both `UNKNOWN` and `FAIL` seem applicable, emit `UNKNOWN` and list the
|
||||
suspected fail condition as a manual review target.
|
||||
@@ -0,0 +1,50 @@
|
||||
# Output Format
|
||||
|
||||
Use Markdown unless the user asks for JSON.
|
||||
|
||||
```markdown
|
||||
# Trailmark Review Gate
|
||||
|
||||
## Verdict
|
||||
|
||||
Gate: PASS | WARN | FAIL | UNKNOWN
|
||||
Confidence: High | Medium | Low
|
||||
|
||||
## Triggered Rules
|
||||
|
||||
| Rule | Verdict | Evidence |
|
||||
|---|---|---|
|
||||
|
||||
## Structural Changes
|
||||
|
||||
| Change | Before | After | Review target |
|
||||
|---|---|---|---|
|
||||
|
||||
## Entrypoint And Reachability Changes
|
||||
|
||||
## Privilege And Taint Changes
|
||||
|
||||
## Blast Radius And Complexity Changes
|
||||
|
||||
## Limitations
|
||||
|
||||
## Recommended Reviewer Actions
|
||||
```
|
||||
|
||||
## Evidence Requirements
|
||||
|
||||
For each triggered rule, include:
|
||||
|
||||
- changed node or edge identifier
|
||||
- source file or symbol when available
|
||||
- entrypoint path when relevant
|
||||
- before/after metric when metric-based
|
||||
- manual review target
|
||||
|
||||
## Wording Requirements
|
||||
|
||||
- Say "gate fired" instead of "vulnerability found".
|
||||
- Say "review target" instead of "exploit path" unless exploitability is
|
||||
separately established.
|
||||
- Say `UNKNOWN` when Trailmark fails or parser support is too incomplete.
|
||||
- Do not claim a `PASS` means the change is secure.
|
||||
@@ -0,0 +1,42 @@
|
||||
# Review Integration
|
||||
|
||||
Use the review gate packet as supporting evidence for a human branch review.
|
||||
It should be attached to, pasted into, or summarized alongside line-level
|
||||
review notes.
|
||||
|
||||
## With Differential Review
|
||||
|
||||
Use `differential-review` for line-level analysis and this skill for structural
|
||||
signals. Recommended order:
|
||||
|
||||
1. Run `differential-review` to identify risky changed files and functions.
|
||||
2. Run `graph-evolution` and `trailmark-review-gate` on the same before/after
|
||||
range.
|
||||
3. Cross-reference `FAIL` and `WARN` rules with changed source lines.
|
||||
4. Add the gate packet to the review notes.
|
||||
5. Treat `PASS` as "no configured graph rule fired", not as approval.
|
||||
|
||||
## With PR Review Processes
|
||||
|
||||
When an engagement has a separate PR review workflow, include:
|
||||
|
||||
- gate verdict
|
||||
- triggered rules table
|
||||
- exact changed nodes and paths
|
||||
- manual reviewer actions
|
||||
- limitations
|
||||
|
||||
Do not use GitHub write actions unless the review process explicitly asks for
|
||||
them. The packet is review evidence, not an automatic merge decision.
|
||||
|
||||
## With Remediation Review
|
||||
|
||||
For a fix commit:
|
||||
|
||||
- compare vulnerable base to the proposed fix
|
||||
- check that affected reachable paths changed as expected
|
||||
- check that no new entrypoint or sensitive-sink path appeared
|
||||
- report `UNKNOWN` if graph evidence cannot confirm the structural change
|
||||
|
||||
This does not replace semantic verification that the original finding was
|
||||
fixed.
|
||||
Reference in New Issue
Block a user