mirror of
https://github.com/trailofbits/skills
synced 2026-06-21 14:12:00 +00:00
docs: address adversarial-verification review feedback
This commit is contained in:
@@ -13,18 +13,19 @@ Dispatch two sub-agents with isolated context — one **advocate**, one **skepti
|
||||
|
||||
**Announce at start:** "I'm using the adversarial-verification skill to stress-test this claim."
|
||||
|
||||
## When to use
|
||||
## When to Use
|
||||
|
||||
| Situation | Use this skill? |
|
||||
|-----------|----------------|
|
||||
| Choosing between 2+ technical approaches | YES |
|
||||
| Verifying a bug finding is real (not false positive) | YES |
|
||||
| Reviewing a design decision before commit | YES |
|
||||
| User asks "is this correct?" on non-trivial claim | YES |
|
||||
| Any claim you're inclined to agree with by default | YES — that's the tell |
|
||||
| Simple factual lookup ("what version is X?") | NO |
|
||||
| Obvious syntax error fix | NO |
|
||||
| User has already made the decision and is executing | NO |
|
||||
- Choosing between 2+ technical approaches
|
||||
- Verifying a bug finding is real (not a false positive)
|
||||
- Reviewing a design decision before commit
|
||||
- User asks "is this correct?" about a non-trivial claim
|
||||
- Any claim you're inclined to agree with by default — that's the tell
|
||||
|
||||
## When NOT to Use
|
||||
|
||||
- Simple factual lookup ("what version is X?")
|
||||
- Obvious syntax error fix
|
||||
- User has already made the decision and is executing
|
||||
|
||||
## The Process
|
||||
|
||||
@@ -85,6 +86,13 @@ Present three things:
|
||||
|
||||
Do NOT dump the raw agent outputs unless the user asks. The verdict is the product.
|
||||
|
||||
## Reference Guide
|
||||
|
||||
- Mode selection quick reference: [references/decision-mode.md](references/decision-mode.md) and [references/proof-mode.md](references/proof-mode.md)
|
||||
- Prompt templates for advocate and skeptic dispatch: [references/prompt-templates.md](references/prompt-templates.md)
|
||||
- Verdict table and recommendation shape: [references/synthesis.md](references/synthesis.md)
|
||||
- Failure modes and recovery patterns: [references/anti-patterns.md](references/anti-patterns.md)
|
||||
|
||||
## Anti-patterns
|
||||
|
||||
See [references/anti-patterns.md](references/anti-patterns.md) for full failure modes. The three most important:
|
||||
|
||||
+1
-1
@@ -16,7 +16,7 @@ Common failure modes that destroy the adversarial value of this skill. Each has
|
||||
|
||||
**Why it happens:** Weak prompt. The agent defaulted to balanced reasoning because nothing stopped it.
|
||||
|
||||
**Fix:** Re-dispatch with a stronger prompt. See [prompt-templates.md](prompt-templates.md#re-dispatching-on-hedged-agents). The key phrase: "Do not acknowledge merit in the opposing position. Do not hedge." Do not accept a hedged response as valid output.
|
||||
**Fix:** Re-dispatch with a stronger prompt. The key phrase: "Do not acknowledge merit in the opposing position. Do not hedge." Do not accept a hedged response as valid output. If you need the full templates, return to [SKILL.md](../SKILL.md) Step 3.
|
||||
|
||||
## 3. Shared context leakage
|
||||
|
||||
|
||||
+1
-1
@@ -10,7 +10,7 @@ Use for approach selection, architecture choice, tool selection, strategy debate
|
||||
- Any comparison between 2+ alternatives
|
||||
- Any design decision before implementation
|
||||
|
||||
If the claim is "this bug is real" or "this vulnerability exists," use [proof-mode.md](proof-mode.md) instead.
|
||||
If the claim is "this bug is real" or "this vulnerability exists," return to [SKILL.md](../SKILL.md) Step 2 and pick Proof mode instead.
|
||||
|
||||
## Required inputs before dispatching
|
||||
|
||||
|
||||
+3
-1
@@ -10,7 +10,7 @@ Use for bug findings, security claims, and any assertion of the form "X is real"
|
||||
- "This finding is valid"
|
||||
- "This behavior is a bug, not by design"
|
||||
|
||||
If the claim is "X is the best approach," use [decision-mode.md](decision-mode.md) instead.
|
||||
If the claim is "X is the best approach," return to [SKILL.md](../SKILL.md) Step 2 and pick Decision mode instead.
|
||||
|
||||
## The structure: N null hypotheses
|
||||
|
||||
@@ -46,6 +46,8 @@ Both agents get the SAME list of null hypotheses. They argue the SAME Ps from op
|
||||
|
||||
## Verdict rule for proof mode
|
||||
|
||||
| Condition | Verdict |
|
||||
|-----------|---------|
|
||||
| All Ps REFUTED by advocate, NOT PROVED by skeptic | **CONFIRMED** — the finding is real |
|
||||
| Any P clearly PROVED by skeptic | **DISMISSED** — the finding is a false positive |
|
||||
| Any P in dispute (both sides plausible) | **UNCERTAIN** — gather more evidence for that specific P before committing |
|
||||
|
||||
+1
-1
@@ -84,4 +84,4 @@ Re-dispatch one or both agents if:
|
||||
- Both agents agreed on something you think is wrong
|
||||
- A key piece of evidence wasn't considered
|
||||
|
||||
Re-run with a tighter prompt. See [prompt-templates.md](prompt-templates.md#re-dispatching-on-hedged-agents).
|
||||
Re-run with a tighter prompt. Use the prompt template reference linked from [SKILL.md](../SKILL.md) Step 3.
|
||||
|
||||
Reference in New Issue
Block a user