Sync metadata-driven parameter decoding build from author

Win32 metadata index (win32json) drives per-call signature lookup so
arguments are captured at their true arity and type, [Out] params are
re-read at the call's return position, and buffers are dumped as bytes.
This commit is contained in:
Xusheng
2026-07-30 15:06:07 -04:00
committed by xusheng
parent f9db02e59e
commit 19dc1581ce
18 changed files with 2048 additions and 49 deletions
+3
View File
@@ -1,3 +1,6 @@
[submodule "capa"]
path = capa
url = https://github.com/HullaBrian/capa
[submodule "win32json"]
path = win32json
url = https://github.com/marlersoft/win32json
+56 -2
View File
@@ -30,6 +30,42 @@ the entire trace. Every time a call occurs, ttd-capa checks if the call target i
with the associate module, function name, parameters, and return value. Additionally, ttd-capa automatically attempts to resolve function parameters as strings,
which has the potential to significantly increase quick wins during malware analysis.
## Metadata-driven parameter decoding
Knowing an API's name is not the same as knowing its signature. Without one, all an extractor can do is grab the four
x64 argument registers and guess at each value - so `CloseHandle(hObject)` gets recorded with four arguments, three of
which are whatever the caller happened to leave in RDX/R8/R9, and any integer that happens to point at printable bytes
gets mistaken for a string.
ttd-capa fixes that with Microsoft's own Win32 API metadata, vendored as the [win32json](https://github.com/marlersoft/win32json)
submodule. `tools/build-win32-index.py` flattens those ~300 JSON files into a compact binary index
(`ttd/data/win32-index.bin`, ~2.7 MB) keyed by export name, which the extractor memory-maps at startup. For every call
whose export is in the index, ttd-capa now knows:
- **the real parameter count**, so exactly that many arguments are captured - including the ones past RCX/RDX/R8/R9
(`CreateProcessW` has ten) and none of the register residue that used to masquerade as arguments
- **each parameter's type**, so `PSTR` is decoded as ANSI and `PWSTR` as UTF-16 rather than by trial, and values typed
as `HANDLE`, enums, or plain integers are never dereferenced at all
- **direction** - `[Out]` parameters are re-read at the call's *return* position, so `lpNumberOfBytesRead` and friends
are rendered filled in. This is something a live debugger can't easily do, and it's where a time-travel trace earns
its keep
- **buffer lengths**, from `MemorySize(BytesParamIndex)` and array count parameters, so `InternetReadFile`'s `lpBuffer`
can be captured at its true length once the callee has written it
- **enum and flag tables**, so `dwCreationDisposition=3` renders as `CREATE_ALWAYS`
The decoded view lands in a new `params` array on each call in the report. The `args` array capa matches against keeps
its original shape - it just has the correct arity now, plus any strings recovered from `[Out]` parameters. Symbolic
flag names are deliberately *not* pushed into `args`, since existing rules match flags numerically.
Coverage is the public Windows SDK, which is what the metadata documents: roughly the top 15% of calls in a typical
trace by volume, but the great majority of the *interesting* ones. Calls with no signature - `ntdll` internals, CRT
helpers like `memset` - fall back to the original four-register heuristic unchanged.
To regenerate the index (only needed after bumping the `win32json` submodule):
```powershell
python tools\build-win32-index.py
```
# Prerequisites
- Windows
- v145 for Microsoft C++ Build Tools
@@ -37,11 +73,18 @@ which has the potential to significantly increase quick wins during malware anal
- Python 3.10+ (for CAPA)
# Building ttd-capa
Clone with submodules (`git clone --recursive`, or `git submodule update --init` in an existing clone) - the
`win32json` submodule supplies the API metadata.
1. Open `ttd/ttdcapa-extract.sln` in Visual Studio
2. Ensure that the required nuget packages (`Microsoft.TimeTravelDebugging.Apis` and `nlohmann.json`) are installed
3. Set the build mode to `x64` and `Release`
4. Navigate to `Build > Build Solution` in order to begin the build
The build copies `ttd/data/win32-index.bin` next to the executable. If it's missing, run
`python tools\build-win32-index.py` to generate it; without it the extractor still runs, but falls back to heuristic
argument capture for every call.
After the build, ttd-capa cannot run properly without Microsoft's `TTDReplay.dll` and `TTDReplayCPU.dll` being in the same directory as `ttdcapa-extract.exe`.
To get those DLLs, ensure you have WinDbg instealled already. Then, run the following PowerShell command to find the DLL location on your system:
@@ -85,7 +128,12 @@ python ttd-capa.py <trace.run> <rules-dir> --sample sample.exe -- -vv
Useful flags:
- `--extractor <path>` - specify a path to the ttd-capa extractor executable
- `--max-calls N` - cap huge traces to certain number of calls
- `--with-stack-args` - captures parameters for function calls that are on the stack (capture more than 5+ parameters for function calls)
- `--with-stack-args` - for calls with *no* Win32 metadata, also grab four stack slots past the register arguments.
Calls we have a signature for always capture their true arity, stack parameters included, so this only affects the
heuristic fallback path
- `--win32-index <path>` - use a specific `win32-index.bin` instead of the one next to the extractor
- `--no-metadata` - disable metadata-driven decoding and use the original four-register heuristic everywhere
- `--max-buffer N` - bytes to keep from any one captured buffer (default 256)
- `--keep-json` - keep the generated JSON report after the Python script runs
## Manual
@@ -129,8 +177,14 @@ python ttd-timeline.py <JSON REPORT PATH> --calls
# Limitations
- Only x64 traces are supported at the moment (no x86 or ARM)
- Argument captures are heuristic, so errors may occur
- Only the functions directly exported by loaded modules are logged in the JSON report
- Argument decoding is exact only for APIs covered by the Win32 metadata (public SDK surface). `ntdll` internals,
undocumented APIs, and CRT helpers fall back to the four-register heuristic, so their arguments may still be wrong
- COM interface methods are not resolved; the metadata has no vtable indices, and the call target is a vtable slot
rather than a named export
- Variadic functions (`printf`-style) have no statically knowable argument count, so only their fixed parameters are
decoded
- Structure parameters are recorded as pointers; fields are not expanded
# Verifying the backend in isolation
`tests/test_ttd_extractor.py` loads a report and dumps every feature per scope - helpful to confirm expected `API`/`Number`/`String` features:
+641
View File
@@ -0,0 +1,641 @@
"""
Flatten the win32json Win32 API metadata into a compact binary index that
ttdcapa-extract loads at startup to decode call parameters.
The metadata (https://github.com/marlersoft/win32json, vendored as the `win32json`
submodule) is ~66 MB of JSON across ~300 files. Parsing that at debug time would be
absurd, so we pre-bake it once: resolve the type graph, classify every parameter
into an x64 ABI slot plus a small "decode kind" the C++ side can switch on, and
write a single mmap-friendly blob.
python tools/build-win32-index.py [win32json/api] [-o ttd/data/win32-index.bin]
Regenerate whenever the win32json submodule is bumped. See
WIN32JSON-TTD-INTEGRATION-NOTES.md for what the metadata does and does not give us
(short version: types and semantics yes, ABI classification is ours -- that's the
`SLOT_*` / `classify_param` logic below).
"""
import io
import os
import sys
import json
import glob
import struct
import argparse
from pathlib import Path
MAGIC = b"W32IDX01"
FORMAT_VERSION = 1
# --- decode kinds; keep in sync with ArgKind in ttd/src/win32meta.hpp ----------
K_UNKNOWN = 0
K_INTEGER = 1
K_BOOL = 2
K_HANDLE = 3
K_ENUM = 4
K_FLOAT = 5
K_DOUBLE = 6
K_ANSI_STRING = 7
K_WIDE_STRING = 8
K_ANSI_BUFFER = 9
K_WIDE_BUFFER = 10
K_BYTE_BUFFER = 11
K_PTR_TO_INT = 12
K_STRUCT_PTR = 13
K_FUNC_PTR = 14
K_GUID = 15
K_POINTER = 16
K_PTR_TO_ANSI_STRING = 17
K_PTR_TO_WIDE_STRING = 18
# --- parameter attribute bits; keep in sync with ParamAttr in win32meta.hpp ---
A_IN = 0x01
A_OUT = 0x02
A_OPTIONAL = 0x04
A_CONST = 0x08
A_RESERVED = 0x10
A_NOT_NUL_TERM = 0x20
A_NULNUL_TERM = 0x40
A_COM_OUT_PTR = 0x80
# --- how a buffer's length is determined; keep in sync with AuxKind -----------
AUX_NONE = 0
AUX_BYTES_FROM_PARAM = 1
AUX_COUNT_FROM_PARAM = 2
AUX_COUNT_CONST = 3
# --- function flags; keep in sync with FuncFlag ------------------------------
F_HIDDEN_RET_PTR = 0x01
F_UNSUPPORTED = 0x02
F_SET_LAST_ERROR = 0x04
NO_ENUM = 0xFFFFFFFF
# x64 sizes for the metadata's primitive names
NATIVE_SIZE = {
"Byte": 1, "SByte": 1, "Boolean": 1,
"Int16": 2, "UInt16": 2, "Char": 2,
"Int32": 4, "UInt32": 4, "Single": 4,
"Int64": 8, "UInt64": 8, "Double": 8, "IntPtr": 8, "UIntPtr": 8,
"Guid": 16,
"Void": 0,
}
NATIVE_INT = {
"Byte", "SByte", "Boolean", "Int16", "UInt16", "Char",
"Int32", "UInt32", "Int64", "UInt64", "IntPtr", "UIntPtr",
}
# NativeTypedefs that are really strings, not opaque handles. BSTR carries a
# FreeFunc (SysFreeString) so the handle heuristic below would otherwise claim it.
STRING_TYPEDEFS = {"PSTR": K_ANSI_STRING, "PWSTR": K_WIDE_STRING, "BSTR": K_WIDE_STRING}
MAX_PARAMS = 255 # param_count is a u8; nothing real comes close (max observed: 18)
class StringTable:
"""Deduplicating NUL-terminated string pool. Offset 0 is always ""."""
def __init__(self):
self.buf = bytearray(b"\x00")
self.offsets = {"": 0}
def add(self, s):
if s is None:
s = ""
off = self.offsets.get(s)
if off is None:
off = len(self.buf)
self.buf += s.encode("utf-8") + b"\x00"
self.offsets[s] = off
return off
class Metadata:
"""The win32json type graph, plus the resolution helpers built on top of it."""
def __init__(self, api_dir):
self.types = {} # (api, name) -> type dict
self.functions = [] # (api, function dict)
self.unicode_aliases = set()
files = sorted(glob.glob(os.path.join(api_dir, "*.json")))
if not files:
sys.exit(f"no .json files under {api_dir}; is the win32json submodule checked out?")
for path in files:
api = os.path.basename(path)[:-5]
with open(path, encoding="utf-8") as f:
doc = json.load(f)
for t in doc.get("Types", []):
# first definition wins; arch-specific duplicates are handled by
# SupportedArchitecture, which we resolve to x64 at index time
self.types.setdefault((api, t["Name"]), t)
for fn in doc.get("Functions", []):
self.functions.append((api, fn))
self.unicode_aliases.update(doc.get("UnicodeAliases", []))
self._size_cache = {}
def lookup(self, ref):
return self.types.get((ref.get("Api"), ref.get("Name")))
def resolve(self, t, depth=0):
"""Follow ApiRef -> NativeTypedef chains to a terminal type node.
Returns (node, typedef) where `typedef` is the last NativeTypedef we passed
through (or None). Callers need the typedef to spot HANDLE/PSTR, which are
only distinguishable by name -- their definitions are plain IntPtr/Byte*.
"""
seen = set()
typedef = None
while depth < 16:
depth += 1
if t.get("Kind") != "ApiRef":
return t, typedef
key = (t.get("Api"), t.get("Name"))
if key in seen:
return t, typedef
seen.add(key)
target = self.lookup(t)
if target is None:
return t, typedef
if target.get("Kind") == "NativeTypedef":
typedef = target
t = target["Def"]
continue
return target, typedef
return t, typedef
# --- sizeof, needed only to classify by-value aggregates ------------------
def sizeof(self, t, depth=0):
"""x64 size in bytes, or None if it can't be determined.
Only aggregates passed/returned by value need this (236 params and 18
return types across the whole surface), so a None here costs us one
function, not correctness everywhere.
"""
if depth > 24:
return None
kind = t.get("Kind")
if kind == "Native":
return NATIVE_SIZE.get(t.get("Name"))
if kind in ("PointerTo", "LPArray", "FunctionPointer"):
return 8
if kind == "Array":
child = self.sizeof(t["Child"], depth + 1)
count = (t.get("Shape") or {}).get("Size")
if child is None or not count:
return None
return child * count
if kind == "ApiRef":
node, _ = self.resolve(t)
if node.get("Kind") == "ApiRef":
return None # unresolvable reference
return self.sizeof(node, depth + 1)
if kind == "Com":
return 8
if kind == "Enum":
return self.enum_width(t)
if kind in ("Struct", "Union"):
return self._sizeof_record(t, depth)
return None
def _sizeof_record(self, t, depth):
key = id(t)
cached = self._size_cache.get(key)
if cached is not None:
return cached[0]
self._size_cache[key] = (None,) # cycle guard
declared = t.get("Size") or 0
if declared:
self._size_cache[key] = (declared,)
return declared
pack = t.get("PackingSize") or 0
offset = 0
max_align = 1
for field in t.get("Fields", []):
fsize = self.sizeof(field["Type"], depth + 1)
if fsize is None:
self._size_cache[key] = (None,)
return None
align = min(fsize if fsize in (1, 2, 4, 8, 16) else 8, pack) if pack else fsize
align = max(1, min(align if align in (1, 2, 4, 8, 16) else 8, 8))
max_align = max(max_align, align)
if t["Kind"] == "Union":
offset = max(offset, fsize)
else:
offset = (offset + align - 1) // align * align + fsize
size = (offset + max_align - 1) // max_align * max_align if offset else 0
self._size_cache[key] = (size,)
return size
def enum_width(self, enum_type):
node = enum_type
if node.get("Kind") == "ApiRef":
node, _ = self.resolve(node)
integer_base = node.get("IntegerBase")
if integer_base:
return NATIVE_SIZE.get(integer_base, 4)
return 4
def attr_bits(attrs):
"""Fold a param's Attrs list into a bitmask plus its MemorySize source, if any."""
bits = 0
bytes_param = None
for a in attrs:
if isinstance(a, dict):
if a.get("Kind") == "MemorySize":
bytes_param = a.get("BytesParamIndex")
continue
bits |= {
"In": A_IN, "Out": A_OUT, "Optional": A_OPTIONAL, "Const": A_CONST,
"Reserved": A_RESERVED, "NotNullTerminated": A_NOT_NUL_TERM,
"NullNullTerminated": A_NULNUL_TERM, "ComOutPtr": A_COM_OUT_PTR,
}.get(a, 0)
return bits, bytes_param
class Param:
__slots__ = ("name", "type_name", "kind", "attrs", "slot",
"aux_kind", "aux_value", "enum_idx", "pointee_size")
def __init__(self):
self.name = ""
self.type_name = ""
self.kind = K_UNKNOWN
self.attrs = 0
self.slot = 0
self.aux_kind = AUX_NONE
self.aux_value = 0
self.enum_idx = NO_ENUM
self.pointee_size = 0
def type_display_name(t):
"""A short human-readable name for the report/timeline, e.g. "PWSTR", "void*"."""
kind = t.get("Kind")
if kind == "ApiRef":
return t.get("Name", "")
if kind == "Native":
return t.get("Name", "")
if kind == "PointerTo":
return type_display_name(t["Child"]) + "*"
if kind == "LPArray":
return type_display_name(t["Child"]) + "[]"
if kind == "FunctionPointer":
return "fnptr"
return kind or ""
def classify_pointee(md, child):
"""Classify what a pointer points at -> (kind, pointee_size).
Split out because PointerTo and LPArray share it.
"""
node, typedef = md.resolve(child)
if typedef is not None and typedef["Name"] in STRING_TYPEDEFS:
# e.g. PWSTR* -- an out-parameter that receives an allocated string
return ({K_ANSI_STRING: K_PTR_TO_ANSI_STRING,
K_WIDE_STRING: K_PTR_TO_WIDE_STRING}[STRING_TYPEDEFS[typedef["Name"]]], 8)
kind = node.get("Kind")
if kind == "Native":
name = node.get("Name")
if name in ("Byte", "SByte"):
return K_ANSI_BUFFER, 1
if name == "Char":
return K_WIDE_BUFFER, 2
if name == "Void":
return K_POINTER, 0
if name == "Guid":
return K_GUID, 16
if name in NATIVE_INT:
return K_PTR_TO_INT, NATIVE_SIZE[name]
if name in ("Single", "Double"):
return K_PTR_TO_INT, NATIVE_SIZE[name]
return K_POINTER, 0
if kind == "Enum":
return K_PTR_TO_INT, md.enum_width(node)
if kind in ("Struct", "Union", "Com"):
return K_STRUCT_PTR, md.sizeof(node) or 0
if kind == "FunctionPointer":
return K_FUNC_PTR, 8
if kind in ("PointerTo", "LPArray"):
return K_PTR_TO_INT, 8 # void** / T** -- at least surface the inner pointer
return K_POINTER, 0
def classify_param(md, p, enum_ids):
"""Map one metadata parameter onto a decode kind + buffer-length source.
Returns a Param with everything except `slot` filled in (slot assignment needs
whole-signature context and happens in build_function).
"""
out = Param()
out.name = p.get("Name") or ""
t = p["Type"]
out.type_name = type_display_name(t)
out.attrs, bytes_param = attr_bits(p.get("Attrs") or [])
kind = t.get("Kind")
if kind == "LPArray":
out.kind, out.pointee_size = classify_pointee(md, t["Child"])
if out.kind == K_POINTER:
out.kind = K_BYTE_BUFFER
count_param = t.get("CountParamIndex", -1)
count_const = t.get("CountConst", -1)
if count_param is not None and count_param >= 0:
out.aux_kind, out.aux_value = AUX_COUNT_FROM_PARAM, count_param
elif count_const is not None and count_const >= 0:
out.aux_kind, out.aux_value = AUX_COUNT_CONST, count_const
elif kind == "PointerTo":
out.kind, out.pointee_size = classify_pointee(md, t["Child"])
elif kind == "ApiRef":
node, typedef = md.resolve(t)
if typedef is not None and typedef["Name"] in STRING_TYPEDEFS:
out.kind = STRING_TYPEDEFS[typedef["Name"]]
out.pointee_size = 1 if out.kind == K_ANSI_STRING else 2
elif typedef is not None and is_handle_typedef(typedef):
# an opaque kernel/GDI/etc handle: pointer-sized, never dereference it
out.kind, out.pointee_size = K_HANDLE, 8
else:
nkind = node.get("Kind")
if nkind == "Enum":
out.kind = K_ENUM
out.pointee_size = md.enum_width(node)
enum_key = (node.get("__api__"), node["Name"])
out.enum_idx = enum_ids.get(enum_key, NO_ENUM)
elif nkind == "Native":
out.kind, out.pointee_size = classify_native(node)
elif nkind in ("Struct", "Union"):
size = md.sizeof(node)
if size in (1, 2, 4, 8):
out.kind, out.pointee_size = K_INTEGER, size
elif size is None:
out.kind = K_UNKNOWN
else:
# x64: aggregates that aren't 1/2/4/8 bytes go by hidden pointer
out.kind, out.pointee_size = K_STRUCT_PTR, size
elif nkind == "Com":
out.kind, out.pointee_size = K_POINTER, 8
elif nkind == "FunctionPointer":
out.kind, out.pointee_size = K_FUNC_PTR, 8
elif nkind == "PointerTo":
out.kind, out.pointee_size = classify_pointee(md, node["Child"])
else:
out.kind = K_UNKNOWN
elif kind == "Native":
out.kind, out.pointee_size = classify_native(t)
if out.kind == K_GUID:
out.kind = K_STRUCT_PTR # 16 bytes by value -> hidden pointer on x64
elif kind == "FunctionPointer":
out.kind, out.pointee_size = K_FUNC_PTR, 8
else:
out.kind = K_UNKNOWN
# a MemorySize attribute always wins: it says this really is a sized buffer
if bytes_param is not None and bytes_param >= 0:
out.aux_kind, out.aux_value = AUX_BYTES_FROM_PARAM, bytes_param
if out.kind in (K_POINTER, K_UNKNOWN, K_STRUCT_PTR):
out.kind = K_BYTE_BUFFER
out.pointee_size = 1
return out
def is_handle_typedef(typedef):
"""Is this NativeTypedef an opaque handle we must never dereference?
RAIIFree/InvalidHandleValue mark most of them (HANDLE, HKEY, ...). The rest are
pointer-sized H-prefixed typedefs with no lifetime metadata (HWND, HDESK, ...);
treating those as integers would be harmless but renders worse.
"""
if typedef.get("FreeFunc") or typedef.get("InvalidHandleValue") is not None:
return True
name = typedef["Name"]
return (
name.startswith("H")
and typedef["Def"].get("Kind") == "Native"
and typedef["Def"].get("Name") in ("IntPtr", "UIntPtr")
)
def classify_native(t):
name = t.get("Name")
if name == "Single":
return K_FLOAT, 4
if name == "Double":
return K_DOUBLE, 8
if name == "Boolean":
return K_BOOL, 1
if name == "Guid":
return K_GUID, 16
if name == "Void":
return K_UNKNOWN, 0
if name in NATIVE_INT:
return K_INTEGER, NATIVE_SIZE[name]
return K_UNKNOWN, 0
def wants_x64(arches):
"""Architectures==[] means arch-neutral; otherwise it must list X64."""
return not arches or "X64" in arches
def build_function(md, api, fn, enum_ids):
"""Produce (params, flags) for one function, or None if it isn't x64-relevant."""
if not wants_x64(fn.get("Architectures") or []):
return None
flags = F_SET_LAST_ERROR if fn.get("SetLastError") else 0
# A function returning an aggregate that isn't 1/2/4/8 bytes takes a hidden
# pointer in RCX, pushing every real parameter one slot to the right.
ret = fn.get("ReturnType") or {}
ret_node, _ = md.resolve(ret) if ret.get("Kind") == "ApiRef" else (ret, None)
base_slot = 0
if ret_node.get("Kind") in ("Struct", "Union") or (
ret_node.get("Kind") == "Native" and ret_node.get("Name") == "Guid"
):
size = md.sizeof(ret_node)
if size is None:
flags |= F_UNSUPPORTED
elif size not in (1, 2, 4, 8):
flags |= F_HIDDEN_RET_PTR
base_slot = 1
raw_params = fn.get("Params") or []
if len(raw_params) + base_slot > MAX_PARAMS:
return None
params = []
for i, p in enumerate(raw_params):
cp = classify_param(md, p, enum_ids)
cp.slot = base_slot + i
if cp.kind == K_UNKNOWN:
flags |= F_UNSUPPORTED
params.append(cp)
# aux_value indexes into the *parameter* list; the C++ side reads captured
# values by parameter index too, so no slot translation is needed. Drop
# references that point outside the list rather than trusting them at runtime.
for cp in params:
if cp.aux_kind in (AUX_BYTES_FROM_PARAM, AUX_COUNT_FROM_PARAM):
if not (0 <= cp.aux_value < len(params)):
cp.aux_kind, cp.aux_value = AUX_NONE, 0
return params, flags
def collect_enums(md):
"""Index only the enums a parameter can actually reference (950 of 7005)."""
referenced = {}
def visit(t, depth=0):
if depth > 16:
return
kind = t.get("Kind")
if kind == "ApiRef":
node, _ = md.resolve(t)
if node.get("Kind") == "Enum":
referenced.setdefault((t.get("Api"), t.get("Name")), node)
elif kind in ("PointerTo", "LPArray"):
visit(t["Child"], depth + 1)
for api, fn in md.functions:
for p in fn.get("Params") or []:
visit(p["Type"])
enums = []
enum_ids = {}
for (eapi, ename), node in sorted(referenced.items()):
node = dict(node)
node["__api__"] = eapi
enum_ids[(eapi, ename)] = len(enums)
enums.append((eapi, ename, node))
return enums, enum_ids
def resolve_enum_index(md, t, enum_ids, depth=0):
"""The enum table index a param type refers to, or NO_ENUM."""
if depth > 16:
return NO_ENUM
kind = t.get("Kind")
if kind == "ApiRef":
node, _ = md.resolve(t)
if node.get("Kind") == "Enum":
return enum_ids.get((t.get("Api"), t.get("Name")), NO_ENUM)
elif kind in ("PointerTo", "LPArray"):
return resolve_enum_index(md, t["Child"], enum_ids, depth + 1)
return NO_ENUM
def build(api_dir, out_path):
print(f"[+] loading {api_dir} ...")
md = Metadata(api_dir)
print(f"[+] {len(md.functions)} functions, {len(md.types)} types")
enums, enum_ids = collect_enums(md)
print(f"[+] {len(enums)} referenced enums")
strtab = StringTable()
# enum tables
enum_recs = []
enumval_recs = []
for eapi, ename, node in enums:
values = node.get("Values") or []
val_off = len(enumval_recs)
for v in values:
enumval_recs.append((strtab.add(v["Name"]), int(v["Value"])))
enum_recs.append((
strtab.add(ename), val_off, len(values),
1 if node.get("Flags") else 0, md.enum_width(node),
))
# functions, keyed by bare export name (module is a weak hint only -- API sets
# mean kernel32!CreateFileW surfaces as KERNELBASE!CreateFileW in a trace)
func_recs = []
param_recs = []
seen_names = {}
skipped_arch = 0
unsupported = 0
collisions = 0
for api, fn in sorted(md.functions, key=lambda x: x[1]["Name"]):
name = fn["Name"]
built = build_function(md, api, fn, enum_ids)
if built is None:
skipped_arch += 1
continue
params, flags = built
if name in seen_names:
prior = seen_names[name]
if prior != tuple((p.kind, p.slot) for p in params):
collisions += 1
continue
seen_names[name] = tuple((p.kind, p.slot) for p in params)
for i, p in enumerate(params):
enum_idx = resolve_enum_index(md, fn["Params"][i]["Type"], enum_ids)
param_recs.append((
strtab.add(p.name), strtab.add(p.type_name), p.kind, p.attrs,
p.slot, p.aux_kind, p.aux_value, enum_idx, p.pointee_size,
))
if flags & F_UNSUPPORTED:
unsupported += 1
func_recs.append((
strtab.add(name), strtab.add(fn.get("DllImport") or ""),
len(param_recs) - len(params), len(params), flags,
))
print(f"[+] indexed {len(func_recs)} functions "
f"({skipped_arch} not x64, {unsupported} flagged UNSUPPORTED, "
f"{collisions} name collisions with differing shapes)")
print(f"[+] {len(param_recs)} params, {len(enumval_recs)} enum values, "
f"{len(strtab.buf)} bytes of strings")
blob = io.BytesIO()
blob.write(MAGIC)
blob.write(struct.pack(
"<IIIIII", FORMAT_VERSION, len(func_recs), len(param_recs),
len(enum_recs), len(enumval_recs), len(strtab.buf),
))
for name_off, dll_off, param_off, count, flags in func_recs:
blob.write(struct.pack("<IIIBBH", name_off, dll_off, param_off, count, flags, 0))
for rec in param_recs:
name_off, type_off, kind, attrs, slot, aux_kind, aux_value, enum_idx, pointee = rec
blob.write(struct.pack("<IIBBBBiIH2x", name_off, type_off, kind, attrs,
slot, aux_kind, aux_value, enum_idx, min(pointee, 0xFFFF)))
for name_off, val_off, val_count, is_flags, width in enum_recs:
blob.write(struct.pack("<IIIBB2x", name_off, val_off, val_count, is_flags, width))
for name_off, value in enumval_recs:
blob.write(struct.pack("<I4xq", name_off, value))
blob.write(strtab.buf)
out_path = Path(out_path)
out_path.parent.mkdir(parents=True, exist_ok=True)
out_path.write_bytes(blob.getvalue())
print(f"[+] wrote {out_path} ({out_path.stat().st_size / 1024 / 1024:.2f} MB)")
return 0
def main(argv):
here = Path(__file__).resolve().parent.parent
ap = argparse.ArgumentParser(description=__doc__.splitlines()[1])
ap.add_argument("api_dir", nargs="?", default=str(here / "win32json" / "api"),
help="path to win32json/api")
ap.add_argument("-o", "--output", default=str(here / "ttd" / "data" / "win32-index.bin"),
help="output blob path")
args = ap.parse_args(argv)
return build(args.api_dir, args.output)
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
+22 -3
View File
@@ -60,7 +60,19 @@ def main(argv: list[str]) -> int:
parser.add_argument("--sample", help="optional on-disk sample for accurate hashes")
parser.add_argument("--extractor", help="path to ttdcapa-extract.exe")
parser.add_argument("--max-calls", type=int, help="cap recorded API calls (for huge traces)")
parser.add_argument("--with-stack-args", action="store_true", help="also capture stack args 5+")
parser.add_argument(
"--with-stack-args",
action="store_true",
help="for calls with no Win32 metadata, also grab four stack slots past the "
"register args (calls we have a signature for always capture their true arity)",
)
parser.add_argument("--win32-index", help="path to win32-index.bin (default: next to the extractor)")
parser.add_argument(
"--no-metadata",
action="store_true",
help="disable metadata-driven argument decoding entirely",
)
parser.add_argument("--max-buffer", type=int, help="bytes kept from any one captured buffer (default 256)")
parser.add_argument("--keep-json", action="store_true", help="keep the intermediate TTD report")
parser.add_argument("--python", default=sys.executable, help="python interpreter to run capa")
args, capa_extra = parser.parse_known_args(argv)
@@ -71,9 +83,10 @@ def main(argv: list[str]) -> int:
trace = Path(args.trace)
if not trace.is_file():
sys.exit(f"trace not found: {trace}")
rules = Path(args.rules)
# absolute, because capa is invoked with cwd set to the report's directory
rules = Path(args.rules).resolve()
if not rules.exists():
sys.exit(f"rules path not found: {rules}")
sys.exit(f"rules path not found: {args.rules}")
extractor = find_extractor(args.extractor)
@@ -89,6 +102,12 @@ def main(argv: list[str]) -> int:
extract_cmd += ["--max-calls", str(args.max_calls)]
if args.with_stack_args:
extract_cmd += ["--with-stack-args"]
if args.win32_index:
extract_cmd += ["--win32-index", args.win32_index]
if args.no_metadata:
extract_cmd += ["--no-metadata"]
if args.max_buffer:
extract_cmd += ["--max-buffer", str(args.max_buffer)]
print(f"[ttd-capa] extracting: {' '.join(extract_cmd)}", file=sys.stderr)
rc = subprocess.call(extract_cmd)
+39 -1
View File
@@ -28,6 +28,20 @@ from typing import Optional
HERE = Path(__file__).resolve().parent
# Recovered guest strings are arbitrary text -- non-Latin paths, HTTP bodies, the
# occasional lone surrogate from a half-written UTF-16 buffer. When stdout is a
# console Python uses UTF-8, but when it is redirected to a file it falls back to
# the locale encoding (cp1252 here), which cannot encode any of that and aborts the
# whole run. Pin UTF-8 and degrade unencodable characters instead of dying.
for _stream in (sys.stdout, sys.stderr):
if hasattr(_stream, "reconfigure"):
_stream.reconfigure(encoding="utf-8", errors="replace")
# Python puts this script's directory first on sys.path, and the capa fork lives in
# a `capa/` subdirectory of it -- which shadows the installed `capa` package with a
# namespace package that has no submodules. Drop it; nothing here is imported locally.
sys.path[:] = [p for p in sys.path if p and Path(p).resolve() != HERE]
from capa.features.address import ProcessAddress, ThreadAddress, DynamicCallAddress # noqa: E402
from capa.features.extractors.ttd.models import TtdCall # noqa: E402
from capa.features.extractors.ttd.extractor import TtdExtractor # noqa: E402
@@ -66,9 +80,33 @@ def fmt_arg(a) -> str:
return repr(a)
def fmt_param(p) -> str:
"""One metadata-decoded parameter as `name=value`.
Prefers the most informative rendering the extractor managed: decoded text,
then symbolic flag names, then a dereferenced pointee, then the raw value.
"""
if p.str_ is not None:
value = repr(p.str_)
elif p.flags:
value = "|".join(p.flags)
elif p.float_ is not None:
value = repr(p.float_)
else:
value = fmt_arg(p.value)
if p.deref is not None:
value += f"->{fmt_arg(p.deref)}"
if p.at_return:
value += "@ret"
return f"{p.name}={value}" if p.name else value
def format_call(call: TtdCall) -> str:
api = f"{call.module}.{call.api}" if call.module else call.api
args = ", ".join(fmt_arg(a) for a in call.args)
if call.params:
args = ", ".join(fmt_param(p) for p in call.params)
else:
args = ", ".join(fmt_arg(a) for a in call.args)
ret = "" if call.ret is None else f" -> 0x{call.ret & 0xFFFFFFFFFFFFFFFF:x}"
return f"{api}({args}){ret}"
Binary file not shown.
+422
View File
@@ -0,0 +1,422 @@
#include "abi_x64.hpp"
#include <cstdio>
#include <cstring>
using ttdcapa::win32meta::ArgKind;
using ttdcapa::win32meta::AuxKind;
namespace ttdcapa {
namespace {
// A dereference is only worth attempting above the first 64 KiB; the null
// page and its neighbours are never mapped in user mode.
constexpr uint64_t kMinDerefAddr = 0x10000;
// Guards against a mis-typed count parameter turning into a huge read.
constexpr uint64_t kMaxCountElements = 1u << 20;
bool readGuest(TTD::Replay::IThreadView const* thread, uint64_t addr, void* dst, size_t size) {
if (addr < kMinDerefAddr || size == 0) {
return false;
}
auto result = thread->QueryMemoryBuffer(TTD::GuestAddress{ addr }, TTD::BufferView{ dst, size });
return result.Memory.Size == size;
}
// The value in `slot`, honouring the shared integer/SSE slot numbering.
uint64_t fetchSlot(const AMD64_CONTEXT& ctx, TTD::Replay::IThreadView const* thread,
uint8_t slot, bool isFloat, bool& ok) {
ok = true;
if (slot < 4) {
if (isFloat) {
const M128BIT* xmm[4] = { &ctx.Xmm0, &ctx.Xmm1, &ctx.Xmm2, &ctx.Xmm3 };
return xmm[slot]->Low;
}
const uint64_t gpr[4] = { ctx.Rcx, ctx.Rdx, ctx.R8, ctx.R9 };
return gpr[slot];
}
// Above the shadow space the caller reserved for RCX/RDX/R8/R9.
uint64_t addr = ctx.Rsp + 0x28 + static_cast<uint64_t>(slot - 4) * 8;
uint64_t v = 0;
ok = readGuest(thread, addr, &v, sizeof(v));
return v;
}
double floatValue(const win32meta::ParamSig& p, uint64_t bits) {
if (p.kind == ArgKind::Float) {
float f = 0.0f;
uint32_t lo = static_cast<uint32_t>(bits);
std::memcpy(&f, &lo, sizeof(f));
return static_cast<double>(f);
}
double d = 0.0;
std::memcpy(&d, &bits, sizeof(d));
return d;
}
// Widen a pointee of `size` bytes to 64 bits.
uint64_t narrowRead(const uint8_t* raw, uint16_t size) {
uint64_t v = 0;
std::memcpy(&v, raw, size > 8 ? 8 : size);
return v;
}
bool derefScalar(TTD::Replay::IThreadView const* thread, uint64_t ptr, uint16_t size, uint64_t& out) {
uint16_t width = size == 0 ? 8 : (size > 8 ? 8 : size);
uint8_t buf[8] = {};
if (!readGuest(thread, ptr, buf, width)) {
return false;
}
out = narrowRead(buf, width);
return true;
}
std::string formatGuid(const uint8_t* b) {
char buf[40];
std::snprintf(buf, sizeof(buf),
"{%08lX-%04X-%04X-%02X%02X-%02X%02X%02X%02X%02X%02X}",
static_cast<unsigned long>(narrowRead(b, 4)),
static_cast<unsigned>(narrowRead(b + 4, 2)),
static_cast<unsigned>(narrowRead(b + 6, 2)),
b[8], b[9], b[10], b[11], b[12], b[13], b[14], b[15]);
return buf;
}
bool isBufferKind(ArgKind kind) {
return kind == ArgKind::AnsiBuffer || kind == ArgKind::WideBuffer || kind == ArgKind::ByteBuffer;
}
// How many bytes a counted buffer parameter spans, or 0 when we can't tell.
// `args` supplies the sibling parameter the count lives in -- after a return
// pass those may themselves have been filled in, which is exactly what makes
// ReadFile's lpBuffer renderable at its *actual* length.
uint64_t resolveByteCount(const win32meta::ParamSig& p, const std::vector<DecodedArg>& args) {
uint64_t count = 0;
switch (p.auxKind) {
case AuxKind::CountConst:
count = static_cast<uint64_t>(p.auxValue);
break;
case AuxKind::BytesFromParam:
case AuxKind::CountFromParam: {
if (p.auxValue < 0 || static_cast<size_t>(p.auxValue) >= args.size()) {
return 0;
}
const DecodedArg& src = args[static_cast<size_t>(p.auxValue)];
count = src.has_deref ? src.deref : src.raw;
break;
}
case AuxKind::None:
default:
return 0;
}
if (count == 0 || count > kMaxCountElements) {
return 0;
}
if (p.auxKind == AuxKind::BytesFromParam) {
return count;
}
uint16_t elem = p.pointeeSize ? p.pointeeSize : 1;
return count * elem;
}
// Length of `buf` up to and including its first `charWidth`-wide NUL. A
// character buffer's count parameter is the caller's *capacity*, so without
// this the report would carry a few hundred bytes of unrelated stack memory
// after every out-string.
size_t terminatorEnd(const std::vector<uint8_t>& buf, size_t charWidth) {
for (size_t i = 0; i + charWidth <= buf.size(); i += charWidth) {
bool nul = true;
for (size_t k = 0; k < charWidth; ++k) {
if (buf[i + k] != 0) {
nul = false;
break;
}
}
if (nul) {
return i + charWidth;
}
}
return buf.size();
}
// Read a counted buffer into `arg`, capped at opt.max_buffer. Character
// buffers additionally get a textual rendering, since that's what a rule or
// an analyst actually wants to see.
void captureBuffer(TTD::Replay::IThreadView const* thread, const DecodeOptions& opt,
ArgKind kind, uint64_t ptr, uint64_t byteCount, DecodedArg& arg) {
if (ptr < kMinDerefAddr || byteCount == 0) {
return;
}
size_t want = static_cast<size_t>(byteCount < opt.max_buffer ? byteCount : opt.max_buffer);
std::vector<uint8_t> buf(want);
auto result = thread->QueryMemoryBuffer(TTD::GuestAddress{ ptr }, TTD::BufferView{ buf.data(), want });
size_t got = result.Memory.Size;
if (got == 0) {
return;
}
buf.resize(got);
if (kind == ArgKind::AnsiBuffer) {
if (auto s = readAnsiString(thread, ptr, got)) {
arg.str = std::move(*s);
arg.has_str = !arg.str.empty();
}
buf.resize(terminatorEnd(buf, 1));
} else if (kind == ArgKind::WideBuffer) {
if (auto s = readWideString(thread, ptr, got / sizeof(wchar_t))) {
arg.str = std::move(*s);
arg.has_str = !arg.str.empty();
}
buf.resize(terminatorEnd(buf, 2));
}
arg.bytes = std::move(buf);
}
// Everything that needs the callee to have run. Shared by the entry pass
// (for [In] parameters, which are already valid) and the return pass.
void dereference(TTD::Replay::IThreadView const* thread, const DecodeOptions& opt,
ArgKind kind, uint64_t ptr, uint16_t pointeeSize,
uint64_t byteCount, DecodedArg& arg) {
switch (kind) {
case ArgKind::AnsiString:
if (auto s = readAnsiString(thread, ptr, opt.max_string)) {
arg.str = std::move(*s);
arg.has_str = true;
}
break;
case ArgKind::WideString:
if (auto s = readWideString(thread, ptr, opt.max_string)) {
arg.str = std::move(*s);
arg.has_str = true;
}
break;
case ArgKind::PtrToInt: {
uint64_t v = 0;
if (derefScalar(thread, ptr, pointeeSize, v)) {
arg.deref = v;
arg.has_deref = true;
}
break;
}
case ArgKind::PtrToAnsiString:
case ArgKind::PtrToWideString: {
uint64_t inner = 0;
if (!derefScalar(thread, ptr, 8, inner)) {
break;
}
arg.deref = inner;
arg.has_deref = true;
auto s = (kind == ArgKind::PtrToAnsiString)
? readAnsiString(thread, inner, opt.max_string)
: readWideString(thread, inner, opt.max_string);
if (s) {
arg.str = std::move(*s);
arg.has_str = true;
}
break;
}
case ArgKind::Guid: {
uint8_t g[16] = {};
if (readGuest(thread, ptr, g, sizeof(g))) {
arg.str = formatGuid(g);
arg.has_str = true;
}
break;
}
case ArgKind::AnsiBuffer:
case ArgKind::WideBuffer:
case ArgKind::ByteBuffer:
captureBuffer(thread, opt, kind, ptr, byteCount, arg);
break;
default:
break;
}
}
// Some parameters are pointers whose pointee the metadata can't pin down:
// opaque void*, and the raw UInt16*/UIntPtr* that RPC uses for RPC_WSTR.
// For those the old guess-if-it-looks-like-text heuristic is still the best
// information available -- and unlike before, we now only apply it to values
// we know really are parameters and really are pointers.
bool mayHoldUntypedString(ArgKind kind) {
return kind == ArgKind::Pointer || kind == ArgKind::Unknown || kind == ArgKind::PtrToInt;
}
void tryUntypedString(TTD::Replay::IThreadView const* thread, DecodedArg& arg) {
if (arg.has_str || !mayHoldUntypedString(arg.kind)) {
return;
}
if (auto s = tryReadString(thread, arg.raw)) {
arg.str = std::move(*s);
arg.has_str = true;
return;
}
// A T** out-parameter: the string lives one more hop away.
if (arg.has_deref) {
if (auto s = tryReadString(thread, arg.deref)) {
arg.str = std::move(*s);
arg.has_str = true;
}
}
}
bool needsDeref(ArgKind kind) {
switch (kind) {
case ArgKind::AnsiString:
case ArgKind::WideString:
case ArgKind::PtrToInt:
case ArgKind::PtrToAnsiString:
case ArgKind::PtrToWideString:
case ArgKind::Guid:
case ArgKind::AnsiBuffer:
case ArgKind::WideBuffer:
case ArgKind::ByteBuffer:
return true;
default:
return false;
}
}
} // namespace
void decodeArgs(const win32meta::FuncSig& sig,
const AMD64_CONTEXT& ctx,
TTD::Replay::IThreadView const* thread,
const DecodeOptions& opt,
std::vector<DecodedArg>& out,
std::vector<PendingOut>& deferred) {
out.clear();
out.resize(sig.paramCount);
// Pass 1: capture every raw slot first. A buffer's length can live in a
// parameter that comes *after* it (ReadFile's lpBuffer refers forward to
// nNumberOfBytesToRead), so no dereferencing until all the scalars are in.
for (uint8_t i = 0; i < sig.paramCount; ++i) {
const win32meta::ParamSig& p = sig.params[i];
DecodedArg& arg = out[i];
arg.name = p.name;
arg.type = p.type;
arg.kind = p.kind;
arg.enum_index = p.enumIndex;
arg.is_out = p.isOut();
bool ok = false;
arg.raw = fetchSlot(ctx, thread, p.slot, p.isFloat(), ok);
if (!ok) {
// Stack slot we couldn't read: leave it zero rather than invent one.
arg.raw = 0;
}
if (p.isFloat()) {
arg.fval = floatValue(p, arg.raw);
arg.has_fval = true;
}
}
// Pass 2: dereference. Scalars and handles are deliberately untouched --
// that alone removes most of the bogus String features the old heuristic
// produced from flag values that happened to look like addresses.
for (uint8_t i = 0; i < sig.paramCount; ++i) {
const win32meta::ParamSig& p = sig.params[i];
DecodedArg& arg = out[i];
if (!needsDeref(p.kind)) {
tryUntypedString(thread, arg);
continue;
}
uint64_t byteCount = isBufferKind(p.kind) ? resolveByteCount(p, out) : 0;
// [In] contents are already valid here. [In,Out] gets read twice: once
// now for what the caller passed, then again at the return.
if (p.isIn()) {
dereference(thread, opt, p.kind, arg.raw, p.pointeeSize, byteCount, arg);
tryUntypedString(thread, arg);
}
if (p.isOut() && arg.raw >= kMinDerefAddr) {
PendingOut pending;
pending.param_index = i;
pending.kind = p.kind;
pending.ptr = arg.raw;
pending.pointee_size = p.pointeeSize;
pending.aux_kind = p.auxKind;
pending.aux_value = p.auxValue;
pending.in_cap = byteCount;
deferred.push_back(pending);
}
}
}
void resolvePendingOuts(const std::vector<PendingOut>& pending,
TTD::Replay::IThreadView const* thread,
const DecodeOptions& opt,
std::vector<DecodedArg>& args) {
// Scalars first: a buffer's real length is usually itself an [Out] scalar
// (ReadFile's lpNumberOfBytesRead), so it has to be resolved before the
// buffer that depends on it.
for (const PendingOut& po : pending) {
if (po.param_index >= args.size() || isBufferKind(po.kind)) {
continue;
}
DecodedArg& arg = args[po.param_index];
DecodedArg fresh;
dereference(thread, opt, po.kind, po.ptr, po.pointee_size, 0, fresh);
if (fresh.has_deref || fresh.has_str) {
fresh.name = arg.name;
fresh.type = arg.type;
fresh.kind = arg.kind;
fresh.enum_index = arg.enum_index;
fresh.raw = arg.raw;
fresh.is_out = true;
fresh.from_return = true;
arg = std::move(fresh);
}
tryUntypedString(thread, arg);
}
for (const PendingOut& po : pending) {
if (po.param_index >= args.size() || !isBufferKind(po.kind)) {
continue;
}
DecodedArg& arg = args[po.param_index];
// Prefer the length the callee reported; fall back to what the caller
// offered, and never exceed it.
win32meta::ParamSig probe;
probe.auxKind = po.aux_kind;
probe.auxValue = po.aux_value;
probe.pointeeSize = po.pointee_size;
uint64_t byteCount = resolveByteCount(probe, args);
if (byteCount == 0) {
byteCount = po.in_cap;
} else if (po.in_cap != 0 && byteCount > po.in_cap) {
byteCount = po.in_cap;
}
if (byteCount == 0) {
continue;
}
DecodedArg fresh;
captureBuffer(thread, opt, po.kind, po.ptr, byteCount, fresh);
if (!fresh.bytes.empty() || fresh.has_str) {
arg.bytes = std::move(fresh.bytes);
arg.str = std::move(fresh.str);
arg.has_str = fresh.has_str;
arg.from_return = true;
}
}
}
std::vector<ArgValue> toCapaArgs(const std::vector<DecodedArg>& args) {
std::vector<ArgValue> out;
out.reserve(args.size());
for (const DecodedArg& a : args) {
if (a.has_str && !a.str.empty()) {
out.push_back(a.str);
} else {
out.push_back(static_cast<int64_t>(a.raw));
}
}
return out;
}
} // namespace ttdcapa
+60
View File
@@ -0,0 +1,60 @@
#ifndef ABI_X64_HPP
#define ABI_X64_HPP
// The x64 calling-convention decoder.
//
// The Win32 metadata gives us types and semantics; mapping those onto registers
// and stack slots is ours to write (WIN32JSON-TTD-INTEGRATION-NOTES.md section 3).
// That's this file: given a signature and a thread's register state at a CALL,
// produce one DecodedArg per real parameter -- no more, no less.
//
// Microsoft x64 in one paragraph: the first four parameters go in RCX/RDX/R8/R9,
// or XMM0-3 if they're floating point. Crucially the slot index is *shared* between
// the two register files, so a float in position 2 lives in XMM2, not XMM0.
// Parameters five and up sit at [RSP+0x28] onwards, above the 32-byte shadow space
// the caller must reserve. Aggregates that aren't exactly 1/2/4/8 bytes are passed
// by hidden pointer, and a function returning one takes an extra hidden first
// parameter -- the indexer pre-computes both, so `slot` here is always final.
#include <cstdint>
#include <vector>
#include <TTD/IReplayEngineStl.h>
#include <TTD/IReplayEngineRegisters.h>
#include "ttdutils.hpp"
#include "win32meta.hpp"
namespace ttdcapa {
struct DecodeOptions {
size_t max_buffer = 256; // bytes of any one counted buffer to keep
size_t max_string = 512; // characters of any one string to keep
};
// Decode every parameter of `sig` from the register state at the call.
// Dereferences that only make sense once the callee has run are appended to
// `deferred` instead; feed those to resolvePendingOuts at the return.
void decodeArgs(const win32meta::FuncSig& sig,
const AMD64_CONTEXT& ctx,
TTD::Replay::IThreadView const* thread,
const DecodeOptions& opt,
std::vector<DecodedArg>& out,
std::vector<PendingOut>& deferred);
// Re-read the deferred dereferences at the return position and patch them into
// `args` (which must be the vector decodeArgs filled in for this same call).
void resolvePendingOuts(const std::vector<PendingOut>& pending,
TTD::Replay::IThreadView const* thread,
const DecodeOptions& opt,
std::vector<DecodedArg>& args);
// Flatten decoded parameters into the int/string list capa matches rules
// against. Strings recovered from [Out] parameters are included -- they are
// genuine new evidence; symbolic enum names are not, so rules that match flags
// numerically keep working.
std::vector<ArgValue> toCapaArgs(const std::vector<DecodedArg>& args);
} // namespace ttdcapa
#endif
+111 -24
View File
@@ -3,7 +3,13 @@
// (capa/features/extractors/ttd/). x64 traces only in v1.
//
// ttdcapa-extract <trace.run> [--sample <sample.exe>] [-o <out.json>]
// [--max-calls N] [--with-stack-args]
// [--max-calls N] [--with-stack-args] [--win32-index <path>]
// [--no-metadata] [--max-buffer N]
// ttdcapa-extract --dump-sig <ApiName>
//
// Arguments are decoded against the pre-baked Win32 metadata index whenever the
// resolved export is in it (see win32meta.hpp / abi_x64.hpp); calls we have no
// signature for fall back to the original heuristic capture.
#include <cassert>
#define DBG_ASSERT(cond) assert(cond)
@@ -30,18 +36,67 @@
#include "ttdutils.hpp"
#include "utils.hpp"
#include "ttd_pe_utils.hpp"
#include "win32meta.hpp"
#include "abi_x64.hpp"
using namespace ttdcapa;
Report g_report;
// Print one function's decoded signature and exit. Lets the metadata index and the
// ABI classification be checked without waiting on a trace replay.
static int dumpSignature(const std::string& api) {
const win32meta::FuncSig* sig = win32meta::index().lookup(api);
if (sig == nullptr) {
std::cerr << "[-] '" << api << "' is not in the metadata index\n";
return 3;
}
std::cout << sig->name << " dll=" << sig->dll
<< " params=" << static_cast<int>(sig->paramCount)
<< (sig->hiddenRetPtr() ? " [hidden-return-pointer]" : "")
<< (sig->unsupported() ? " [unsupported]" : "") << "\n";
for (uint8_t i = 0; i < sig->paramCount; ++i) {
const win32meta::ParamSig& p = sig->params[i];
std::cout << " slot " << static_cast<int>(p.slot) << " " << p.name
<< " : " << p.type << " (" << win32meta::kindName(p.kind) << ")";
if (p.isIn()) std::cout << " in";
if (p.isOut()) std::cout << " out";
if (p.attrs & win32meta::AttrOptional) std::cout << " optional";
if (p.auxKind == win32meta::AuxKind::BytesFromParam) std::cout << " bytes=param[" << p.auxValue << "]";
if (p.auxKind == win32meta::AuxKind::CountFromParam) std::cout << " count=param[" << p.auxValue << "]";
if (p.auxKind == win32meta::AuxKind::CountConst) std::cout << " count=" << p.auxValue;
if (p.hasEnum()) std::cout << " enum=" << win32meta::index().enumName(p.enumIndex);
std::cout << "\n";
}
return 0;
}
int wmain(int argc, wchar_t** argv) {
Options opt;
if (!parse_args(argc, argv, opt)) {
std::cerr << "Usage: ttdcapa-extract <trace.run> [--sample <sample.exe>] [-o <out.json>] [--max-calls N] [--with-stack-args]\n";
std::cerr << "Usage: ttdcapa-extract <trace.run> [--sample <sample.exe>] [-o <out.json>]\n"
" [--max-calls N] [--with-stack-args]\n"
" [--win32-index <path>] [--no-metadata] [--max-buffer N]\n"
" ttdcapa-extract --dump-sig <ApiName>\n";
return 1;
}
DecodeOptions decode_opt;
decode_opt.max_buffer = opt.max_buffer;
if (!opt.no_metadata) {
std::string err;
if (win32meta::loadIndex(opt.win32_index, err)) {
std::cerr << "[+] Loaded Win32 metadata for " << win32meta::index().functionCount()
<< " functions\n";
} else {
std::cerr << "[!] " << err << "; falling back to heuristic argument capture\n";
}
}
if (!opt.dump_sig.empty()) {
return dumpSignature(opt.dump_sig);
}
auto [engine, hr] = TTD::Replay::MakeReplayEngine();
if (hr != S_OK || !engine) {
std::cerr << "[-] Failed to create replay engine: 0x" << std::hex << hr << "\n";
@@ -71,7 +126,8 @@ int wmain(int argc, wchar_t** argv) {
// Get list of function VAs used for call sweep
std::unordered_map<uint64_t, std::pair<std::wstring, std::string>> resolvedTraceModuleExports;
resolvedTraceModuleExports = resolveTraceModuleExports(engine, inspection_cursor);
std::cerr << "[+] Resolved all exported module functions across execution\n";
std::cerr << "[+] Resolved " << resolvedTraceModuleExports.size()
<< " exported module functions across execution\n";
size_t thread_count = engine->GetThreadCount();
TTD::Replay::ThreadInfo const* thread_list = engine->GetThreadList();
@@ -79,9 +135,16 @@ int wmain(int argc, wchar_t** argv) {
g_report.process.threads.push_back(static_cast<uint64_t>(thread_list[i].UniqueId));
}
// per-thread stack of in-flight recorded calls: (expected return addr, call index)
std::unordered_map<uint64_t, std::vector<std::pair<uint64_t, size_t>>> in_flight;
// per-thread stack of recorded calls we haven't seen return yet, each carrying
// the [Out] dereferences we deliberately postponed until the callee has run
struct InFlight {
uint64_t ret_addr = 0;
size_t call_index = 0;
std::vector<PendingOut> pending;
};
std::unordered_map<uint64_t, std::vector<InFlight>> in_flight;
uint64_t seq = 0;
uint64_t events_seen = 0;
bool limit_hit = false;
TTD::Replay::UniqueCursor sweep{ engine->NewCursor() };
@@ -90,6 +153,7 @@ int wmain(int argc, wchar_t** argv) {
TTD::Replay::IThreadView const* thread) noexcept {
bool is_call = (static_cast<uint64_t>(fall_through) != 0);
uint64_t utid = static_cast<uint64_t>(thread->GetThreadInfo().UniqueId);
++events_seen;
if (is_call) {
auto it = resolvedTraceModuleExports.find(static_cast<uint64_t>(target));
@@ -120,33 +184,51 @@ int wmain(int argc, wchar_t** argv) {
TTD::Replay::RegisterContext regs = thread->GetCrossPlatformContext();
auto const* ctx = reinterpret_cast<AMD64_CONTEXT const*>(&regs);
rec.args.push_back(captureCallArg(thread, ctx->Rcx));
rec.args.push_back(captureCallArg(thread, ctx->Rdx));
rec.args.push_back(captureCallArg(thread, ctx->R8));
rec.args.push_back(captureCallArg(thread, ctx->R9));
if (opt.with_stack_args) { // Capture arguments on stack based on offsets from RSP
for (int k = 0; k < 4; ++k) {
uint64_t slot = ctx->Rsp + 0x28 + static_cast<uint64_t>(k) * 8;
uint64_t v = 0;
if (thread->QueryMemoryBuffer(TTD::GuestAddress{ slot }, TTD::BufferView{ &v, sizeof(v) })
.Memory.Size == sizeof(v)) {
rec.args.push_back(captureCallArg(thread, v));
std::vector<PendingOut> pending;
win32meta::FuncSig const* sig = win32meta::index().lookup(rec.api);
if (sig != nullptr && !sig->unsupported()) {
// We know the real arity, so capture exactly that many arguments
// -- no stale RDX/R8/R9 residue masquerading as parameters.
decodeArgs(*sig, *ctx, thread, decode_opt, rec.params, pending);
rec.args = toCapaArgs(rec.params);
rec.metadata = true;
} else {
rec.args.push_back(captureCallArg(thread, ctx->Rcx));
rec.args.push_back(captureCallArg(thread, ctx->Rdx));
rec.args.push_back(captureCallArg(thread, ctx->R8));
rec.args.push_back(captureCallArg(thread, ctx->R9));
if (opt.with_stack_args) { // Capture arguments on stack based on offsets from RSP
for (int k = 0; k < 4; ++k) {
uint64_t slot = ctx->Rsp + 0x28 + static_cast<uint64_t>(k) * 8;
uint64_t v = 0;
if (thread->QueryMemoryBuffer(TTD::GuestAddress{ slot }, TTD::BufferView{ &v, sizeof(v) })
.Memory.Size == sizeof(v)) {
rec.args.push_back(captureCallArg(thread, v));
}
}
}
}
size_t idx = g_report.process.calls.size();
in_flight[utid].push_back({ static_cast<uint64_t>(fall_through), idx });
in_flight[utid].push_back(InFlight{ static_cast<uint64_t>(fall_through), idx, std::move(pending) });
g_report.process.calls.push_back(std::move(rec));
} else {
// Log most recent function called as returned and capture return value
auto it = in_flight.find(utid);
if (it != in_flight.end() && !it->second.empty()) {
auto& top = it->second.back();
if (top.first == static_cast<uint64_t>(target)) {
g_report.process.calls[top.second].ret = thread->GetBasicReturnValue();
g_report.process.calls[top.second].has_ret = true;
InFlight& top = it->second.back();
if (top.ret_addr == static_cast<uint64_t>(target)) {
CallRecord& call = g_report.process.calls[top.call_index];
call.ret = thread->GetBasicReturnValue();
call.has_ret = true;
if (!top.pending.empty()) {
// The payoff of a time-travel trace: [Out] parameters can be
// rendered filled in, which a live debugger can't easily do.
resolvePendingOuts(top.pending, thread, decode_opt, call.params);
call.args = toCapaArgs(call.params);
}
it->second.pop_back();
}
}
@@ -154,7 +236,11 @@ int wmain(int argc, wchar_t** argv) {
};
sweep->SetCallReturnCallback(on_call_return);
sweep->SetReplayFlags(TTD::Replay::ReplayFlags::ReplaySegmentsSequentially);
// Without ReplayAllSegmentsWithoutFiltering the engine only replays segments it
// thinks can hit an event, and a call/return callback alone doesn't qualify --
// the sweep then completes instantly having seen nothing.
sweep->SetReplayFlags(TTD::Replay::ReplayFlags::ReplaySegmentsSequentially |
TTD::Replay::ReplayFlags::ReplayAllSegmentsWithoutFiltering);
sweep->SetPosition(TTD::Replay::Position::Min);
std::cerr << "[+] Beginning execution sweep...\n";
sweep->ReplayForward();
@@ -162,7 +248,8 @@ int wmain(int argc, wchar_t** argv) {
if (limit_hit) {
std::cerr << "[!] Reached --max-calls limit (" << opt.max_calls << ")\n";
}
std::cerr << "[+] Recorded " << g_report.process.calls.size() << " API calls\n";
std::cerr << "[+] Recorded " << g_report.process.calls.size() << " API calls from "
<< events_seen << " call/return events\n";
writeReport(opt.output);
+117 -1
View File
@@ -48,13 +48,30 @@ namespace ttdcapa {
return resolvedTraceModuleExports;
}
std::string convertWstringToString(const std::wstring& ws) {
if (ws.empty()) {
return {};
}
int needed = ::WideCharToMultiByte(CP_UTF8, 0, ws.data(), static_cast<int>(ws.size()), nullptr, 0, nullptr, nullptr);
if (needed <= 0) {
return {};
}
std::string out(static_cast<size_t>(needed), '\0');
::WideCharToMultiByte(CP_UTF8, 0, ws.data(), static_cast<int>(ws.size()), out.data(), needed, nullptr, nullptr);
return out;
}
size_t readMemory(TTD::Replay::UniqueCursor* cursor, TTD::GuestAddress addr, void* dest, unsigned __int64 size) {
TTD::Replay::MemoryBuffer memoryBuffer = cursor->get()->QueryMemoryBuffer(addr, TTD::BufferView{ dest, size });
return memoryBuffer.Memory.Size;
}
bool initializeTTDEngine(TTD::Replay::UniqueReplayEngine& engine, std::wstring traceFilePath) {
ConsoleErrorReporting reporter;
// The engine holds this pointer for its whole lifetime and calls into it
// whenever it reports an error. A stack local would dangle the moment this
// function returns, and the first derailment during ReplayForward would then
// dispatch a virtual call through reclaimed stack memory.
static ConsoleErrorReporting reporter;
engine->RegisterDebugModeAndLogging(TTD::Replay::DebugModeType::None, &reporter);
if (!engine->Initialize(traceFilePath.c_str())) {
@@ -132,6 +149,105 @@ namespace ttdcapa {
return std::nullopt;
}
namespace {
// Shared by both typed readers: an address below the first 64 KiB is never a
// valid user-mode string pointer, and treating one as such is how a flags
// DWORD ends up rendered as text.
constexpr uint64_t kMinStringAddr = 0x10000;
bool isPlausibleTextByte(unsigned char c) {
return c == '\t' || c == '\r' || c == '\n' || (c >= 0x20 && c != 0x7f);
}
// The -A entry points take strings in the ANSI code page, so their high
// bytes are not UTF-8. Emitting them raw would produce a report that the
// JSON serializer refuses to write, so transcode before anything else sees
// them. Pure-ASCII input (the overwhelming majority) short-circuits.
std::string ansiToUtf8(std::string bytes) {
bool ascii = true;
for (unsigned char c : bytes) {
if (c >= 0x80) {
ascii = false;
break;
}
}
if (ascii) {
return bytes;
}
int wide = ::MultiByteToWideChar(CP_ACP, 0, bytes.data(), static_cast<int>(bytes.size()), nullptr, 0);
if (wide > 0) {
std::wstring ws(static_cast<size_t>(wide), L'\0');
if (::MultiByteToWideChar(CP_ACP, 0, bytes.data(), static_cast<int>(bytes.size()), ws.data(), wide) == wide) {
return convertWstringToString(ws);
}
}
// Unconvertible: keep the ASCII skeleton rather than dropping the string.
for (char& c : bytes) {
if (static_cast<unsigned char>(c) >= 0x80) {
c = '?';
}
}
return bytes;
}
} // namespace
std::optional<std::string> readAnsiString(TTD::Replay::IThreadView const* thread, uint64_t addr, size_t maxChars) {
if (addr < kMinStringAddr) {
return std::nullopt;
}
std::vector<char> buf(maxChars + 1);
auto result = thread->QueryMemoryBuffer(TTD::GuestAddress{ addr }, TTD::BufferView{ buf.data(), maxChars });
size_t avail = result.Memory.Size;
if (avail == 0) {
return std::nullopt;
}
std::string s;
for (size_t i = 0; i < avail; ++i) {
unsigned char c = static_cast<unsigned char>(buf[i]);
if (c == 0) {
return ansiToUtf8(std::move(s));
}
if (!isPlausibleTextByte(c)) {
return std::nullopt; // control bytes mean this wasn't a string after all
}
s.push_back(static_cast<char>(c));
}
// Ran out of readable memory before the terminator; keep what we have as
// long as it looked like text the whole way.
return s.empty() ? std::nullopt : std::optional<std::string>(ansiToUtf8(std::move(s)));
}
std::optional<std::string> readWideString(TTD::Replay::IThreadView const* thread, uint64_t addr, size_t maxChars) {
if (addr < kMinStringAddr) {
return std::nullopt;
}
std::vector<wchar_t> buf(maxChars + 1);
auto result = thread->QueryMemoryBuffer(
TTD::GuestAddress{ addr }, TTD::BufferView{ buf.data(), maxChars * sizeof(wchar_t) });
size_t availChars = result.Memory.Size / sizeof(wchar_t);
if (availChars == 0) {
return std::nullopt;
}
size_t len = 0;
while (len < availChars && buf[len] != L'\0') {
wchar_t wc = buf[len];
// Reject C0 controls (other than the usual whitespace) rather than
// emitting mojibake for a pointer that only looked like a string.
if (wc < 0x20 && wc != L'\t' && wc != L'\r' && wc != L'\n') {
return std::nullopt;
}
++len;
}
if (len == 0) {
return std::string{};
}
return convertWstringToString(std::wstring(buf.data(), len));
}
// Capture one candidate argument: a dereferenced string if it points to one, otherwise the raw integer value.
ttdcapa::ArgValue captureCallArg(TTD::Replay::IThreadView const* thread, uint64_t value) {
if (auto s = tryReadString(thread, value)) {
+49 -1
View File
@@ -14,6 +14,8 @@
#include <TTD/IReplayEngineStl.h>
#include <TTD/ErrorReporting.h>
#include "win32meta.hpp"
class ConsoleErrorReporting : public TTD::ErrorReporting {
public:
void __fastcall VPrintError(char const* fmt, va_list args) override {
@@ -43,13 +45,49 @@ namespace ttdcapa {
// One call argument: either an integer or a dereferenced string
using ArgValue = std::variant<int64_t, std::string>;
// One parameter decoded with the help of the Win32 metadata index. `name` and
// `type` point into the index blob, which outlives every record, so they cost
// nothing to copy. Enum values keep their index rather than their decoded flag
// names so millions of in-memory calls stay cheap; names are resolved once at
// report-write time.
struct DecodedArg {
const char* name = nullptr;
const char* type = nullptr;
win32meta::ArgKind kind = win32meta::ArgKind::Unknown;
uint32_t enum_index = 0xFFFFFFFFu;
uint64_t raw = 0; // the register/stack value as captured
uint64_t deref = 0; // pointee, for PtrToInt and friends
double fval = 0.0; // for Float/Double params (read from XMM)
std::string str; // decoded string contents
std::vector<uint8_t> bytes; // bounded buffer preview
bool has_str = false;
bool has_deref = false;
bool has_fval = false;
bool is_out = false;
bool from_return = false; // contents were read at the return position
};
// A dereference deferred until the call returns, so [Out] parameters can be
// rendered filled in -- something only a time-travel trace makes easy.
struct PendingOut {
uint16_t param_index = 0;
win32meta::ArgKind kind = win32meta::ArgKind::Unknown;
uint64_t ptr = 0;
uint16_t pointee_size = 0;
win32meta::AuxKind aux_kind = win32meta::AuxKind::None;
int32_t aux_value = 0;
uint64_t in_cap = 0; // caller-supplied upper bound on length, 0 if unknown
};
struct CallRecord {
uint64_t tid = 0; // TTD Thread ID
uint64_t seq = 0; // monotonic record order (consistent with timeline order)
std::string position; // TTD navigable position "Sequence:Steps" (hex), for WinDbg
std::wstring module; // resolved owning module, e.g. "kernel32" (no extension)
std::string api; // resolved export name, e.g. "CreateFileA"
std::vector<ArgValue> args;
std::vector<ArgValue> args; // flat view consumed by capa (ints and strings)
std::vector<DecodedArg> params; // rich view; only meaningful when `metadata` is set
bool metadata = false; // args came from a real signature, not the heuristic
bool has_ret = false;
uint64_t ret = 0;
};
@@ -78,12 +116,22 @@ namespace ttdcapa {
// TTD memory read utility function
size_t readMemory(TTD::Replay::UniqueCursor* cursor, TTD::GuestAddress addr, void* dest, unsigned __int64 size);
// UTF-16 -> UTF-8, for the wide strings TTD and the PE headers hand back
std::string convertWstringToString(const std::wstring& ws);
// Initializes the TTD engine based off a given trace file (.run) path
bool initializeTTDEngine(TTD::Replay::UniqueReplayEngine& engine, std::wstring trace_file_path);
// Attempts to interpret the memory at a certain address as a string. If not a string, will return null
std::optional<std::string> tryReadString(TTD::Replay::IThreadView const* thread, uint64_t addr);
// Read a NUL-terminated string the metadata told us is really there. Unlike
// tryReadString these do not guess: no minimum length, and the wide reader
// converts real UTF-16 (not just its ASCII subset) to UTF-8. Returns nullopt
// only when the memory is unreadable or the bytes aren't a plausible string.
std::optional<std::string> readAnsiString(TTD::Replay::IThreadView const* thread, uint64_t addr, size_t maxChars = 512);
std::optional<std::string> readWideString(TTD::Replay::IThreadView const* thread, uint64_t addr, size_t maxChars = 512);
// Attempts to capture an argument as a string. If it doesn't look like a valid string, this function will return the same argument value
ArgValue captureCallArg(TTD::Replay::IThreadView const* thread, uint64_t value);
}
+62 -14
View File
@@ -56,18 +56,6 @@ namespace ttdcapa {
return result;
}
std::string convertWstringToString(const std::wstring& ws) {
if (ws.empty()) {
return {};
}
int needed = ::WideCharToMultiByte(CP_UTF8, 0, ws.data(), static_cast<int>(ws.size()), nullptr, 0, nullptr, nullptr);
if (needed <= 0) {
return {};
}
std::string out(static_cast<size_t>(needed), '\0');
::WideCharToMultiByte(CP_UTF8, 0, ws.data(), static_cast<int>(ws.size()), out.data(), needed, nullptr, nullptr);
return out;
}
} // namespace
SampleHashes hashFile(const std::filesystem::path& path) {
@@ -178,6 +166,18 @@ namespace ttdcapa {
else if (a == L"--with-stack-args") {
opt.with_stack_args = true;
}
else if (a == L"--win32-index" && i + 1 < argc) {
opt.win32_index = argv[++i];
}
else if (a == L"--no-metadata") {
opt.no_metadata = true;
}
else if (a == L"--max-buffer" && i + 1 < argc) {
opt.max_buffer = static_cast<size_t>(std::wcstoull(argv[++i], nullptr, 10));
}
else if (a == L"--dump-sig" && i + 1 < argc) {
opt.dump_sig = convertWstringToString(argv[++i]);
}
else if (!a.empty() && a[0] == L'-') {
std::cerr << "unknown option\n";
return false;
@@ -190,7 +190,8 @@ namespace ttdcapa {
return false;
}
}
return !opt.trace.empty();
// --dump-sig is a metadata-only debug mode, so it doesn't need a trace.
return !opt.trace.empty() || !opt.dump_sig.empty();
}
bool writeReport(std::filesystem::path outputFilePath) {
@@ -239,6 +240,9 @@ namespace ttdcapa {
call["module"] = moduleStr;
call["api"] = callRecord.api;
// Always an array, even when empty -- a zero-parameter function is a
// real result, not a missing field.
call["args"] = json::array();
for (ArgValue& argValue : callRecord.args) {
if (std::holds_alternative<int64_t>(argValue)) {
call["args"].push_back(std::get<int64_t>(argValue));
@@ -247,6 +251,46 @@ namespace ttdcapa {
}
}
// The rich, metadata-derived view. capa matches on "args"; this is for
// ttd-timeline and human triage, so it can afford to be verbose. Absent
// entirely when we had no signature, so consumers can tell "no
// parameters" apart from "we didn't know".
if (callRecord.metadata) {
call["params"] = json::array();
}
for (DecodedArg& decoded : callRecord.params) {
json param;
param["name"] = decoded.name ? decoded.name : "";
param["type"] = decoded.type ? decoded.type : "";
param["kind"] = win32meta::kindName(decoded.kind);
param["value"] = decoded.raw;
if (decoded.has_str) {
param["str"] = decoded.str;
}
if (decoded.has_deref) {
param["deref"] = decoded.deref;
}
if (decoded.has_fval) {
param["float"] = decoded.fval;
}
if (decoded.enum_index != 0xFFFFFFFFu) {
std::vector<std::string> names = win32meta::index().decodeEnum(decoded.enum_index, decoded.raw);
if (!names.empty()) {
param["flags"] = names;
}
}
if (!decoded.bytes.empty()) {
param["bytes"] = to_hex(decoded.bytes);
}
if (decoded.is_out) {
param["out"] = true;
}
if (decoded.from_return) {
param["at_return"] = true;
}
call["params"].push_back(std::move(param));
}
call["ret"] = callRecord.has_ret ? callRecord.ret : 0;
process["calls"].push_back(call);
@@ -260,8 +304,12 @@ namespace ttdcapa {
return false;
}
reportFile << report.dump() << std::endl;
// Recovered guest memory can always surprise us with a byte sequence that
// isn't valid UTF-8. Substituting it beats throwing away an entire trace's
// worth of extraction over one bad string.
reportFile << report.dump(-1, ' ', false, nlohmann::json::error_handler_t::replace) << std::endl;
reportFile.close();
std::cerr << "DONE!\n";
return true;
}
}
+9 -3
View File
@@ -31,9 +31,15 @@ namespace ttdcapa {
struct Options {
std::filesystem::path trace;
std::filesystem::path sample; // optional on-disk sample for hashing
std::filesystem::path output; // empty will output to stdout
uint64_t max_calls = 0; // 0 means unlimited
std::filesystem::path sample; // optional on-disk sample for hashing
std::filesystem::path output; // empty will output to stdout
std::filesystem::path win32_index; // empty means search the default locations
std::string dump_sig; // print one signature and exit; no trace needed
uint64_t max_calls = 0; // 0 means unlimited
size_t max_buffer = 256; // bytes kept from any one counted buffer
bool no_metadata = false; // force the pre-metadata heuristic capture
// Only affects calls with no metadata: blindly grab four extra stack slots.
// Functions we have a signature for always capture their true arity.
bool with_stack_args = false;
};
+277
View File
@@ -0,0 +1,277 @@
#include "win32meta.hpp"
#ifndef WIN32_LEAN_AND_MEAN
#define WIN32_LEAN_AND_MEAN
#endif
#include <windows.h>
#include <algorithm>
#include <cstring>
#include <fstream>
namespace ttdcapa::win32meta {
namespace {
constexpr char kMagic[8] = { 'W', '3', '2', 'I', 'D', 'X', '0', '1' };
constexpr uint32_t kFormatVersion = 1;
// Record sizes must match the struct.pack formats in tools/build-win32-index.py.
constexpr size_t kHeaderSize = 32;
constexpr size_t kFuncRecSize = 16;
constexpr size_t kParamRecSize = 24;
constexpr size_t kEnumRecSize = 16;
constexpr size_t kEnumValRecSize = 16;
template <typename T>
T readAt(const uint8_t* p, size_t off) {
T v{};
std::memcpy(&v, p + off, sizeof(T));
return v;
}
int popcount64(uint64_t v) {
int n = 0;
while (v) {
v &= v - 1;
++n;
}
return n;
}
} // namespace
bool Index::load(const std::filesystem::path& path, std::string& error) {
std::ifstream f(path, std::ios::binary);
if (!f) {
error = "cannot open index";
return false;
}
std::vector<uint8_t> blob((std::istreambuf_iterator<char>(f)), std::istreambuf_iterator<char>());
if (blob.size() < kHeaderSize || std::memcmp(blob.data(), kMagic, sizeof(kMagic)) != 0) {
error = "not a win32 index file (bad magic)";
return false;
}
const uint8_t* p = blob.data();
uint32_t version = readAt<uint32_t>(p, 8);
if (version != kFormatVersion) {
error = "index format version " + std::to_string(version) +
", expected " + std::to_string(kFormatVersion) + " (regenerate with tools/build-win32-index.py)";
return false;
}
uint32_t funcCount = readAt<uint32_t>(p, 12);
uint32_t paramCount = readAt<uint32_t>(p, 16);
uint32_t enumCount = readAt<uint32_t>(p, 20);
uint32_t enumValCount = readAt<uint32_t>(p, 24);
uint32_t strtabSize = readAt<uint32_t>(p, 28);
size_t funcOff = kHeaderSize;
size_t paramOff = funcOff + static_cast<size_t>(funcCount) * kFuncRecSize;
size_t enumOff = paramOff + static_cast<size_t>(paramCount) * kParamRecSize;
size_t enumValOff = enumOff + static_cast<size_t>(enumCount) * kEnumRecSize;
size_t strOff = enumValOff + static_cast<size_t>(enumValCount) * kEnumValRecSize;
if (strOff + strtabSize != blob.size()) {
error = "index is truncated or corrupt";
return false;
}
// Every string offset is dereferenced without a bounds check below, so the
// pool must be NUL-terminated for that to be safe.
if (strtabSize == 0 || blob[blob.size() - 1] != 0) {
error = "index string pool is not NUL-terminated";
return false;
}
blob_ = std::move(blob);
p = blob_.data();
const char* strtab = reinterpret_cast<const char*>(p + strOff);
auto str = [&](uint32_t off) -> const char* {
return off < strtabSize ? strtab + off : "";
};
params_.resize(paramCount);
for (uint32_t i = 0; i < paramCount; ++i) {
size_t o = paramOff + static_cast<size_t>(i) * kParamRecSize;
ParamSig& ps = params_[i];
ps.name = str(readAt<uint32_t>(p, o));
ps.type = str(readAt<uint32_t>(p, o + 4));
ps.kind = static_cast<ArgKind>(p[o + 8]);
ps.attrs = p[o + 9];
ps.slot = p[o + 10];
ps.auxKind = static_cast<AuxKind>(p[o + 11]);
ps.auxValue = readAt<int32_t>(p, o + 12);
ps.enumIndex = readAt<uint32_t>(p, o + 16);
ps.pointeeSize = readAt<uint16_t>(p, o + 20);
if (ps.enumIndex != 0xFFFFFFFFu && ps.enumIndex >= enumCount) {
ps.enumIndex = 0xFFFFFFFFu;
}
}
funcs_.resize(funcCount);
byName_.reserve(funcCount * 2);
for (uint32_t i = 0; i < funcCount; ++i) {
size_t o = funcOff + static_cast<size_t>(i) * kFuncRecSize;
FuncSig& fs = funcs_[i];
fs.name = str(readAt<uint32_t>(p, o));
fs.dll = str(readAt<uint32_t>(p, o + 4));
uint32_t firstParam = readAt<uint32_t>(p, o + 8);
fs.paramCount = p[o + 12];
fs.flags = p[o + 13];
if (static_cast<size_t>(firstParam) + fs.paramCount > params_.size()) {
error = "index parameter range out of bounds";
return false;
}
fs.params = fs.paramCount ? &params_[firstParam] : nullptr;
byName_.emplace(std::string_view(fs.name), i);
}
enumValues_.resize(enumValCount);
for (uint32_t i = 0; i < enumValCount; ++i) {
size_t o = enumValOff + static_cast<size_t>(i) * kEnumValRecSize;
enumValues_[i].name = str(readAt<uint32_t>(p, o));
enumValues_[i].value = readAt<int64_t>(p, o + 8);
}
enums_.resize(enumCount);
for (uint32_t i = 0; i < enumCount; ++i) {
size_t o = enumOff + static_cast<size_t>(i) * kEnumRecSize;
EnumTable& et = enums_[i];
et.name = str(readAt<uint32_t>(p, o));
et.valueOffset = readAt<uint32_t>(p, o + 4);
et.valueCount = readAt<uint32_t>(p, o + 8);
et.isFlags = p[o + 12] != 0;
et.width = p[o + 13];
if (static_cast<size_t>(et.valueOffset) + et.valueCount > enumValues_.size()) {
et.valueOffset = 0;
et.valueCount = 0;
}
}
path_ = path;
return true;
}
const FuncSig* Index::lookup(std::string_view api) const {
auto it = byName_.find(api);
return it == byName_.end() ? nullptr : &funcs_[it->second];
}
const char* Index::enumName(uint32_t enumIndex) const {
return enumIndex < enums_.size() ? enums_[enumIndex].name : "";
}
std::vector<std::string> Index::decodeEnum(uint32_t enumIndex, uint64_t value) const {
std::vector<std::string> out;
if (enumIndex >= enums_.size()) {
return out;
}
const EnumTable& et = enums_[enumIndex];
// The captured value is a full 64-bit register; mask to the enum's real width
// so sign-extension and upper garbage don't defeat the comparisons.
uint64_t mask = et.width >= 8 ? ~0ull : ((1ull << (et.width * 8)) - 1);
uint64_t v = value & mask;
const EnumValue* vals = enumValues_.data() + et.valueOffset;
for (uint32_t i = 0; i < et.valueCount; ++i) {
if ((static_cast<uint64_t>(vals[i].value) & mask) == v) {
out.emplace_back(vals[i].name);
return out; // exact match wins, flags or not
}
}
if (!et.isFlags || v == 0) {
return out;
}
// Greedy decomposition: consume the widest matching bit groups first so
// composites like GENERIC_WRITE beat their individual constituent bits.
std::vector<uint32_t> order(et.valueCount);
for (uint32_t i = 0; i < et.valueCount; ++i) {
order[i] = i;
}
std::sort(order.begin(), order.end(), [&](uint32_t a, uint32_t b) {
return popcount64(static_cast<uint64_t>(vals[a].value) & mask) >
popcount64(static_cast<uint64_t>(vals[b].value) & mask);
});
uint64_t remaining = v;
for (uint32_t i : order) {
uint64_t bits = static_cast<uint64_t>(vals[i].value) & mask;
if (bits != 0 && (remaining & bits) == bits) {
out.emplace_back(vals[i].name);
remaining &= ~bits;
}
}
if (remaining != 0) {
char buf[32];
std::snprintf(buf, sizeof(buf), "0x%llx", static_cast<unsigned long long>(remaining));
out.emplace_back(buf);
}
return out;
}
Index& index() {
static Index instance;
return instance;
}
namespace {
std::filesystem::path executableDir() {
wchar_t buf[MAX_PATH * 4];
DWORD n = ::GetModuleFileNameW(nullptr, buf, static_cast<DWORD>(std::size(buf)));
if (n == 0 || n >= std::size(buf)) {
return {};
}
return std::filesystem::path(buf, buf + n).parent_path();
}
} // namespace
bool loadIndex(const std::filesystem::path& explicitPath, std::string& error) {
std::vector<std::filesystem::path> candidates;
if (!explicitPath.empty()) {
candidates.push_back(explicitPath);
} else {
std::filesystem::path dir = executableDir();
if (!dir.empty()) {
candidates.push_back(dir / L"win32-index.bin");
// running straight out of ttd\bin\<plat>\<config>\ during development
candidates.push_back(dir / L".." / L".." / L".." / L"data" / L"win32-index.bin");
}
}
std::error_code ec;
for (const auto& c : candidates) {
if (!std::filesystem::exists(c, ec)) {
continue;
}
if (index().load(c, error)) {
return true;
}
return false; // found but unusable: surface the real reason
}
error = "win32-index.bin not found (run tools/build-win32-index.py, or pass --win32-index)";
return false;
}
const char* kindName(ArgKind kind) {
switch (kind) {
case ArgKind::Integer: return "int";
case ArgKind::Bool: return "bool";
case ArgKind::Handle: return "handle";
case ArgKind::Enum: return "enum";
case ArgKind::Float: return "float";
case ArgKind::Double: return "double";
case ArgKind::AnsiString: return "str";
case ArgKind::WideString: return "wstr";
case ArgKind::AnsiBuffer: return "strbuf";
case ArgKind::WideBuffer: return "wstrbuf";
case ArgKind::ByteBuffer: return "buf";
case ArgKind::PtrToInt: return "int*";
case ArgKind::StructPtr: return "struct*";
case ArgKind::FuncPtr: return "fnptr";
case ArgKind::Guid: return "guid";
case ArgKind::Pointer: return "ptr";
case ArgKind::PtrToAnsiString: return "str*";
case ArgKind::PtrToWideString: return "wstr*";
case ArgKind::Unknown:
default: return "unknown";
}
}
} // namespace ttdcapa::win32meta
+156
View File
@@ -0,0 +1,156 @@
#ifndef WIN32META_HPP
#define WIN32META_HPP
// Loader for the pre-baked Win32 API metadata index (win32-index.bin) produced by
// tools/build-win32-index.py from the win32json submodule.
//
// The index answers one question per recorded call: "given this export name, what
// are its parameters?" -- count, names, types, direction, and how to decode each
// one. That turns the extractor's blind 4-register grab into an exact capture.
//
// Lookup is keyed on the bare function name. Module is carried for display only:
// API sets mean the metadata's "CreateFileW -> KERNEL32.dll" shows up in a trace as
// KERNELBASE.dll!CreateFileW, so matching on module would lose most calls.
// See WIN32JSON-TTD-INTEGRATION-NOTES.md section 4.
#include <cstdint>
#include <filesystem>
#include <string>
#include <string_view>
#include <unordered_map>
#include <vector>
namespace ttdcapa::win32meta {
// How to decode one parameter. Values must match the K_* constants in
// tools/build-win32-index.py.
enum class ArgKind : uint8_t {
Unknown = 0,
Integer, // plain scalar, never dereferenced
Bool,
Handle, // opaque HANDLE/HKEY/HWND; pointer-sized but not a pointer
Enum, // scalar with a symbolic value table (see enumIndex)
Float, // 4-byte float, arrives in XMM<slot>
Double, // 8-byte float, arrives in XMM<slot>
AnsiString, // char*, NUL-terminated
WideString, // wchar_t*, NUL-terminated
AnsiBuffer, // char[], length from aux
WideBuffer, // wchar_t[], length from aux
ByteBuffer, // void*/byte[], length from aux
PtrToInt, // pointer to a pointeeSize-byte scalar
StructPtr, // pointer to a struct/union (not expanded in v1)
FuncPtr,
Guid, // pointer to a 16-byte GUID
Pointer, // opaque pointer, contents unknown
PtrToAnsiString, // char** -- out-param that receives an allocated string
PtrToWideString, // wchar_t**
};
// Bitmask of a parameter's direction/nullability attributes.
enum ParamAttr : uint8_t {
AttrIn = 0x01,
AttrOut = 0x02,
AttrOptional = 0x04,
AttrConst = 0x08,
AttrReserved = 0x10,
AttrNotNulTerminated = 0x20,
AttrNulNulTerminated = 0x40,
AttrComOutPtr = 0x80,
};
// Where a buffer parameter's length comes from.
enum class AuxKind : uint8_t {
None = 0,
BytesFromParam, // auxValue is the index of a parameter holding a byte count
CountFromParam, // auxValue is the index of a parameter holding an element count
CountConst, // auxValue is the element count itself
};
enum FuncFlag : uint8_t {
FlagHiddenRetPtr = 0x01, // returns a large aggregate; RCX is the hidden return buffer
FlagUnsupported = 0x02, // at least one parameter could not be classified
FlagSetLastError = 0x04,
};
struct ParamSig {
const char* name = "";
const char* type = "";
ArgKind kind = ArgKind::Unknown;
uint8_t attrs = 0;
uint8_t slot = 0; // positional ABI slot, already shifted for FlagHiddenRetPtr
AuxKind auxKind = AuxKind::None;
int32_t auxValue = 0;
uint32_t enumIndex = 0xFFFFFFFFu;
uint16_t pointeeSize = 0; // bytes per pointee/element, 0 if unknown
bool isIn() const { return (attrs & AttrIn) != 0; }
bool isOut() const { return (attrs & AttrOut) != 0; }
bool isFloat() const { return kind == ArgKind::Float || kind == ArgKind::Double; }
bool hasEnum() const { return enumIndex != 0xFFFFFFFFu; }
};
struct FuncSig {
const char* name = "";
const char* dll = "";
const ParamSig* params = nullptr;
uint8_t paramCount = 0;
uint8_t flags = 0;
bool unsupported() const { return (flags & FlagUnsupported) != 0; }
bool hiddenRetPtr() const { return (flags & FlagHiddenRetPtr) != 0; }
};
// Loaded once at startup and read-only thereafter, so the replay sweep can hit
// it from the call callback without synchronisation.
class Index {
public:
bool load(const std::filesystem::path& path, std::string& error);
bool loaded() const { return !funcs_.empty(); }
size_t functionCount() const { return funcs_.size(); }
const std::filesystem::path& path() const { return path_; }
// Bare export name, e.g. "CreateFileW". Returns nullptr when unknown, which
// is the caller's cue to fall back to the heuristic capture.
const FuncSig* lookup(std::string_view api) const;
const char* enumName(uint32_t enumIndex) const;
// Symbolic names for `value`: an exact match if one exists, otherwise a
// greedy bit decomposition for flag enums. Empty when nothing matches.
std::vector<std::string> decodeEnum(uint32_t enumIndex, uint64_t value) const;
private:
struct EnumTable {
const char* name = "";
uint32_t valueOffset = 0;
uint32_t valueCount = 0;
bool isFlags = false;
uint8_t width = 4;
};
struct EnumValue {
const char* name = "";
int64_t value = 0;
};
std::vector<uint8_t> blob_;
std::vector<FuncSig> funcs_;
std::vector<ParamSig> params_;
std::vector<EnumTable> enums_;
std::vector<EnumValue> enumValues_;
std::unordered_map<std::string_view, uint32_t> byName_;
std::filesystem::path path_;
};
// Process-wide instance; empty until loadIndex() succeeds.
Index& index();
// Try `explicitPath` if non-empty, else the conventional locations next to the
// executable and in the source tree. Returns false with `error` set; the caller
// is expected to warn and continue in heuristic mode.
bool loadIndex(const std::filesystem::path& explicitPath, std::string& error);
const char* kindName(ArgKind kind);
} // namespace ttdcapa::win32meta
#endif
+11
View File
@@ -70,15 +70,19 @@
</Link>
</ItemDefinitionGroup>
<ItemGroup>
<ClCompile Include="src\abi_x64.cpp" />
<ClCompile Include="src\main.cpp" />
<ClCompile Include="src\ttd_pe_utils.cpp" />
<ClCompile Include="src\ttdutils.cpp" />
<ClCompile Include="src\utils.cpp" />
<ClCompile Include="src\win32meta.cpp" />
</ItemGroup>
<ItemGroup>
<ClInclude Include="src\abi_x64.hpp" />
<ClInclude Include="src\ttd_pe_utils.hpp" />
<ClInclude Include="src\ttdutils.hpp" />
<ClInclude Include="src\utils.hpp" />
<ClInclude Include="src\win32meta.hpp" />
</ItemGroup>
<ItemGroup>
<None Include="packages.config" />
@@ -101,6 +105,13 @@
</ItemGroup>
<Copy SourceFiles="@(TTDRuntime)" DestinationFolder="$(OutDir)" SkipUnchangedFiles="true" Condition="'@(TTDRuntime)'!=''" ContinueOnError="true" />
</Target>
<!-- The Win32 metadata index the argument decoder reads at startup. Regenerate
with tools\build-win32-index.py after bumping the win32json submodule; the
extractor still runs (heuristically) if it is absent. -->
<Target Name="CopyWin32Index" AfterTargets="Build">
<Copy SourceFiles="data\win32-index.bin" DestinationFolder="$(OutDir)" SkipUnchangedFiles="true" Condition="Exists('data\win32-index.bin')" />
<Warning Condition="!Exists('data\win32-index.bin')" Text="data\win32-index.bin is missing; run 'python tools\build-win32-index.py' to enable metadata-driven argument decoding." />
</Target>
<Import Project="packages\nlohmann.json.3.12.0\build\native\nlohmann.json.targets" Condition="Exists('packages\nlohmann.json.3.12.0\build\native\nlohmann.json.targets')" />
<Target Name="EnsureNuGetPackageBuildImports" BeforeTargets="PrepareForBuild">
<PropertyGroup>
+12
View File
@@ -1,9 +1,15 @@
<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<ItemGroup>
<ClCompile Include="src\abi_x64.cpp">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="src\main.cpp">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="src\win32meta.cpp">
<Filter>Source Files</Filter>
</ClCompile>
<ClCompile Include="src\ttd_pe_utils.cpp">
<Filter>Source Files</Filter>
</ClCompile>
@@ -15,9 +21,15 @@
</ClCompile>
</ItemGroup>
<ItemGroup>
<ClInclude Include="src\abi_x64.hpp">
<Filter>Header Files</Filter>
</ClInclude>
<ClInclude Include="src\ttd_pe_utils.hpp">
<Filter>Header Files</Filter>
</ClInclude>
<ClInclude Include="src\win32meta.hpp">
<Filter>Header Files</Filter>
</ClInclude>
<ClInclude Include="src\ttdutils.hpp">
<Filter>Header Files</Filter>
</ClInclude>
Submodule
+1
Submodule win32json added at 071df49952