Files
trailofbits-skills/plugins/trailmark/skills/graph-evolution/scripts/graph_diff.py
T
Scott Arciszewski 6b79b5e879 feat(trailmark): skills that reason about code as graphs (#133)
* feat(trailmark): skills that reason about code as graphs

* Add Codex skill symlinks for trailmark plugin

The trailmark plugin's 10 skills were missing .codex/skills/ mappings,
which caused the validate_codex_skills CI check to fail.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Address PR #133 review feedback

- Add Rationalizations (Do Not Skip) sections to 5 security skills:
  trailmark, audit-augmentation, crypto-protocol-diagram,
  mermaid-to-proverif, graph-evolution
- Fix requires-python: diagram.py >= 3.12 (was 3.13), protocol.py
  >= 3.12 (was 3.11) to match trailmark's actual requirement
- Rename diagram/ to diagramming-code/ to match SKILL.md frontmatter
  name and all cross-skill references; update Codex symlink

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Fix diagram skill to use uv run instead of plain python

The diagram.py script carries PEP 723 inline metadata declaring
trailmark as a dependency. Plain python ignores this metadata,
causing ImportError for users who haven't pre-installed trailmark.
uv run processes the metadata and handles dependency resolution.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Address second round of PR #133 review feedback

- Fix README directory tree: diagram/ -> diagramming-code/
- Fix diagram-types.md: python -> uv run for all script invocations
- Fix graph-evolution Phase 3: replace undefined shell variables
  ($BEFORE_JSON etc) with template substitutions ({before_json} etc)
- Fix vector-forge mutation-frameworks.md: replace cross-skill file
  link with prose reference to genotoxic skill (avoids reference chain)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Local skill-improver review pass across all 10 trailmark skills

diagramming-code:
- Fix arrow syntax inconsistency: uncertain edges use ..-> not -.->
- Fix extra closing paren in diagram-types.md
- Fix diagram.py docstring to match uv run invocation

crypto-protocol-diagram:
- Remove reference chain: spec-parsing-patterns.md no longer links to
  mermaid-sequence-syntax.md, inlines the arrow syntax instead
- Fix ProVerif example note: "Tamarin/ProVerif" -> "ProVerif"

trailmark:
- Replace "path/to/project" with {targetDir} in query-patterns.md
- Add uv run prefix to CLI examples in query-patterns.md
- Add circom to supported language list
- Add pre-analysis annotation kinds to annotation docs

genotoxic:
- Remove reference chains: triage-methodology.md and
  mutation-frameworks.md no longer link to graph-analysis.md

vector-forge:
- Add trailmark to Prerequisites section
- Fix bare trailmark commands to use uv run with {targetDir}

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Address third round of PR #133 review feedback

mermaid-to-proverif:
- Fix ProVerif type error: verify(...) = true is a type mismatch
  since verify returns bitstring. Use let _ = verify(...) in instead,
  which aborts on destructor failure (correct ProVerif pattern)

trailmark-summary, trailmark-structural:
- Add 8 missing language extensions to find command (.rb, .php, .cs,
  .java, .hs, .erl, .cairo, .circom)
- Remove unsupported .lean extension
- Split .c -> --language c and .cpp -> --language cpp (separate parsers)

All 7 security skills:
- Rename "Rationalizations (Do Not Skip)" to "Rationalizations to
  Reject" per CLAUDE.md convention

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Address fourth round of PR #133 review feedback

mermaid-to-proverif:
- Fix ProVerif type errors in process template: pkey values cannot
  appear in bitstring positions. Add pkey2bs() and concat() to the
  function declarations and rewrite the template to use them,
  matching the sample-output.pv example

trailmark-summary:
- Split .js/.ts mapping: .js -> --language javascript,
  .ts -> --language typescript (separate parsers)

graph-evolution:
- Replace bare python with python3 in graph_diff.py invocations
  (python does not exist on modern Ubuntu/Debian/macOS)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Address fifth round of PR #133 review feedback

graph-evolution:
- Change python3 to uv run for graph_diff.py invocations to match
  ecosystem convention

trailmark-structural, trailmark-summary:
- Add Rationalizations to Reject sections (both are security skills
  running blast radius, taint, and privilege boundary analysis)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Fix ProVerif type consistency and graph-evolution template vars

mermaid-to-proverif:
- Rename senc/sdec to aead_enc/aead_dec in Step 3 preamble to match
  the process template and sample-output.pv
- Fix hkdf signature: hkdf(key, bitstring): key (first arg is DH
  shared secret which has type key, not bitstring)

crypto-to-proverif-mapping.md:
- Fix hkdf declaration and summary table to match corrected signature
- Fix example to use concat/pkey2bs for type-correct HKDF input

graph-evolution:
- Replace $BEFORE_DIR/$AFTER_DIR shell vars in Phase 5 with
  {before_dir}/{after_dir} template substitutions

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Comprehensive vivisect-style review of all trailmark skills

ProVerif correctness (mermaid-to-proverif):
- Fix broken ForwardSecrecyTest pattern in security-properties.md:
  process waited on c_fs but nothing sent on it, past_session_key
  was never bound to any session. Replaced with working pattern
  that leaks long-term keys and checks session key secrecy.
- Fix hkdf(bitstring,bitstring) -> hkdf(key,bitstring) in
  proverif-syntax.md to match SKILL.md and sample-output.pv
- Fix type-incorrect example in proverif-syntax.md: tuple of
  (key,pkey,pkey) passed where bitstring expected. Now uses
  concat2/pkey2bs for type-correct serialization.
- Align senc/sdec -> aead_enc/aead_dec in proverif-syntax.md and
  crypto-to-proverif-mapping.md to match SKILL.md and example
- Fix auth query parameter count in security-properties.md:
  beginI fires before session key is known, so has fewer params

Cross-skill consistency:
- Fix 3 stale "diagram skill" references -> "diagramming-code"
  in trailmark/SKILL.md and preanalysis-passes.md
- Add PEP 723 header to graph_diff.py for convention consistency

README and helper skills:
- Add trailmark-summary and trailmark-structural to README skills
  table and directory tree
- Add secondary file extensions (.jsx, .tsx, .h, .hpp, .cc, .cxx)
  to language detection in summary and structural skills
- Inline language mapping in trailmark-structural (was deferred
  to trailmark-summary, violating one-level-deep rule)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Fix ProVerif type consistency and graph-evolution template vars

- Fix endInitiator -> endI in mermaid-to-proverif Step 6 template
  (endInitiator was never declared as an event)
- Add missing msg2_label constant to Step 3 constants block
- Add .hh/.hxx C++ header extensions to language detection in
  trailmark-summary and trailmark-structural

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Fix mermaid-to-proverif template: missing beginI event and secrecy witness

Step 6 Initiator template:
- Add missing event beginI(pk(sk_I), pk_R) before first out —
  without it, authentication queries always report false attacks
- Replace local new secret_I with free private_I [private] to
  match sample-output.pv's secrecy witness pattern

security-properties.md:
- Fix beginI/beginR from 3 args to 2 args in mutual auth section
  and query checklist (begin events fire before session key is
  known, so they only take the two public keys)
- Update "Placing Events" table to match 2-param form

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Address sixth round of PR #133 review feedback

proverif-syntax.md Two-Party Process example:
- Fix type errors: pkey values passed directly to bitstring params
  in sign() and verify(). Now uses concat2(pkey2bs(...)) pattern.
- Add missing pkey2bs declaration to function list
- Add missing info_session constant declaration
- Fix msg2_label -> msg2 in verification check example to match
  the file's own constant declarations

trailmark-structural:
- Fix contradiction: Rationalizations table said "Install trailmark
  first" but Execution section forbids install commands. Changed
  to "Report not installed and return" to match execution policy.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 17:31:30 -04:00

276 lines
7.7 KiB
Python

# /// script
# requires-python = ">=3.12"
# ///
"""Compute structural diff between two Trailmark graph JSON exports.
Compares nodes, edges, complexity, subgraph membership, and
pre-analysis results to surface security-relevant structural changes.
"""
from __future__ import annotations
import argparse
import json
import sys
from pathlib import Path
from typing import Any
def load_graph(path: str) -> dict[str, Any]:
"""Load and validate a Trailmark JSON export."""
data = json.loads(Path(path).read_text())
for key in ("nodes", "edges"):
if key not in data:
print(f"ERROR: Missing '{key}' in {path}", file=sys.stderr)
sys.exit(1)
return data
def diff_nodes(
before: dict[str, Any],
after: dict[str, Any],
) -> dict[str, Any]:
"""Compute added, removed, and modified nodes."""
before_ids = set(before.keys())
after_ids = set(after.keys())
added = _summarize_nodes(after, after_ids - before_ids)
removed = _summarize_nodes(before, before_ids - after_ids)
modified = _find_modified(before, after, before_ids & after_ids)
return {"added": added, "removed": removed, "modified": modified}
def _summarize_nodes(
nodes: dict[str, Any],
ids: set[str],
) -> list[dict[str, Any]]:
"""Extract summary dicts for a set of node IDs."""
result = []
for nid in sorted(ids):
node = nodes[nid]
result.append(
{
"id": nid,
"name": node.get("name", ""),
"kind": node.get("kind", ""),
"file": _node_file(node),
"cyclomatic_complexity": node.get("cyclomatic_complexity"),
}
)
return result
def _node_file(node: dict[str, Any]) -> str:
"""Extract file path from a node's location."""
loc = node.get("location", {})
if isinstance(loc, dict):
return loc.get("file_path", "")
return ""
def _find_modified(
before: dict[str, Any],
after: dict[str, Any],
shared_ids: set[str],
) -> list[dict[str, Any]]:
"""Find nodes present in both with changed properties."""
modified = []
for nid in sorted(shared_ids):
b, a = before[nid], after[nid]
changes = _compare_node_properties(b, a)
if changes:
modified.append({"id": nid, "changes": changes})
return modified
def _compare_node_properties(
before: dict[str, Any],
after: dict[str, Any],
) -> dict[str, Any]:
"""Compare security-relevant properties of two node versions."""
changes: dict[str, Any] = {}
cc_b = before.get("cyclomatic_complexity")
cc_a = after.get("cyclomatic_complexity")
if cc_b != cc_a:
changes["cyclomatic_complexity"] = {
"before": cc_b,
"after": cc_a,
}
params_b = _param_signature(before)
params_a = _param_signature(after)
if params_b != params_a:
changes["parameters"] = {
"before": params_b,
"after": params_a,
}
ret_b = _return_type_str(before)
ret_a = _return_type_str(after)
if ret_b != ret_a:
changes["return_type"] = {"before": ret_b, "after": ret_a}
span_b = _line_span(before)
span_a = _line_span(after)
if span_b != span_a:
changes["line_span"] = {"before": span_b, "after": span_a}
return changes
def _param_signature(node: dict[str, Any]) -> list[str]:
"""Extract parameter names from a node."""
params = node.get("parameters", ())
if isinstance(params, (list, tuple)):
return [p.get("name", "") if isinstance(p, dict) else str(p) for p in params]
return []
def _return_type_str(node: dict[str, Any]) -> str | None:
"""Extract return type string from a node."""
rt = node.get("return_type")
if isinstance(rt, dict):
return rt.get("name")
return rt
def _line_span(node: dict[str, Any]) -> int:
"""Compute line count from a node's location."""
loc = node.get("location", {})
if isinstance(loc, dict):
start = loc.get("start_line", 0)
end = loc.get("end_line", 0)
return max(0, end - start + 1)
return 0
def diff_edges(
before: list[dict[str, Any]],
after: list[dict[str, Any]],
) -> dict[str, Any]:
"""Compute added and removed edges."""
before_set = {_edge_key(e) for e in before}
after_set = {_edge_key(e) for e in after}
added = sorted(after_set - before_set)
removed = sorted(before_set - after_set)
return {
"added": [_parse_edge_key(k) for k in added],
"removed": [_parse_edge_key(k) for k in removed],
}
def _edge_key(edge: dict[str, Any]) -> str:
"""Create a hashable key for an edge."""
src = edge.get("source", edge.get("source_id", ""))
tgt = edge.get("target", edge.get("target_id", ""))
kind = edge.get("kind", "")
return f"{src}|{tgt}|{kind}"
def _parse_edge_key(key: str) -> dict[str, str]:
"""Convert an edge key back to a dict."""
source, target, kind = key.split("|", 2)
return {"source": source, "target": target, "kind": kind}
def diff_subgraphs(
before: dict[str, list[str]],
after: dict[str, list[str]],
) -> dict[str, Any]:
"""Compute per-subgraph membership changes."""
all_names = sorted(set(before.keys()) | set(after.keys()))
changes: dict[str, Any] = {}
for name in all_names:
b_ids = set(before.get(name, []))
a_ids = set(after.get(name, []))
added = sorted(a_ids - b_ids)
removed = sorted(b_ids - a_ids)
if added or removed:
changes[name] = {"added": added, "removed": removed}
return changes
def compute_summary_delta(
before: dict[str, Any],
after: dict[str, Any],
) -> dict[str, Any]:
"""Compute deltas for summary statistics."""
b_sum = before.get("summary", {})
a_sum = after.get("summary", {})
delta: dict[str, Any] = {}
for key in ("total_nodes", "functions", "classes", "call_edges", "entrypoints"):
b_val = b_sum.get(key, 0)
a_val = a_sum.get(key, 0)
if b_val != a_val:
delta[key] = {
"before": b_val,
"after": a_val,
"delta": a_val - b_val,
}
return delta
def compute_diff(
before: dict[str, Any],
after: dict[str, Any],
) -> dict[str, Any]:
"""Compute the full structural diff between two graphs."""
return {
"summary_delta": compute_summary_delta(before, after),
"nodes": diff_nodes(
before.get("nodes", {}),
after.get("nodes", {}),
),
"edges": diff_edges(
before.get("edges", []),
after.get("edges", []),
),
"subgraphs": diff_subgraphs(
before.get("subgraphs", {}),
after.get("subgraphs", {}),
),
}
def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
"""Parse command-line arguments."""
parser = argparse.ArgumentParser(
description="Structural diff between Trailmark graphs",
)
parser.add_argument(
"--before",
required=True,
help="Path to the 'before' graph JSON export",
)
parser.add_argument(
"--after",
required=True,
help="Path to the 'after' graph JSON export",
)
parser.add_argument(
"--indent",
type=int,
default=2,
help="JSON output indentation (default: 2)",
)
return parser.parse_args(argv)
def main(argv: list[str] | None = None) -> None:
"""Entry point: load graphs, compute diff, print JSON."""
args = parse_args(argv)
before = load_graph(args.before)
after = load_graph(args.after)
diff = compute_diff(before, after)
print(json.dumps(diff, indent=args.indent))
if __name__ == "__main__":
main()