diff --git a/.gitmodules b/.gitmodules index 90236b6..68bed30 100644 --- a/.gitmodules +++ b/.gitmodules @@ -1,3 +1,6 @@ [submodule "capa"] path = capa url = https://github.com/HullaBrian/capa +[submodule "win32json"] + path = win32json + url = https://github.com/marlersoft/win32json diff --git a/README.md b/README.md index f69cf15..0fb1101 100644 --- a/README.md +++ b/README.md @@ -30,6 +30,42 @@ the entire trace. Every time a call occurs, ttd-capa checks if the call target i with the associate module, function name, parameters, and return value. Additionally, ttd-capa automatically attempts to resolve function parameters as strings, which has the potential to significantly increase quick wins during malware analysis. +## Metadata-driven parameter decoding +Knowing an API's name is not the same as knowing its signature. Without one, all an extractor can do is grab the four +x64 argument registers and guess at each value - so `CloseHandle(hObject)` gets recorded with four arguments, three of +which are whatever the caller happened to leave in RDX/R8/R9, and any integer that happens to point at printable bytes +gets mistaken for a string. + +ttd-capa fixes that with Microsoft's own Win32 API metadata, vendored as the [win32json](https://github.com/marlersoft/win32json) +submodule. `tools/build-win32-index.py` flattens those ~300 JSON files into a compact binary index +(`ttd/data/win32-index.bin`, ~2.7 MB) keyed by export name, which the extractor memory-maps at startup. For every call +whose export is in the index, ttd-capa now knows: + +- **the real parameter count**, so exactly that many arguments are captured - including the ones past RCX/RDX/R8/R9 + (`CreateProcessW` has ten) and none of the register residue that used to masquerade as arguments +- **each parameter's type**, so `PSTR` is decoded as ANSI and `PWSTR` as UTF-16 rather than by trial, and values typed + as `HANDLE`, enums, or plain integers are never dereferenced at all +- **direction** - `[Out]` parameters are re-read at the call's *return* position, so `lpNumberOfBytesRead` and friends + are rendered filled in. This is something a live debugger can't easily do, and it's where a time-travel trace earns + its keep +- **buffer lengths**, from `MemorySize(BytesParamIndex)` and array count parameters, so `InternetReadFile`'s `lpBuffer` + can be captured at its true length once the callee has written it +- **enum and flag tables**, so `dwCreationDisposition=3` renders as `CREATE_ALWAYS` + +The decoded view lands in a new `params` array on each call in the report. The `args` array capa matches against keeps +its original shape - it just has the correct arity now, plus any strings recovered from `[Out]` parameters. Symbolic +flag names are deliberately *not* pushed into `args`, since existing rules match flags numerically. + +Coverage is the public Windows SDK, which is what the metadata documents: roughly the top 15% of calls in a typical +trace by volume, but the great majority of the *interesting* ones. Calls with no signature - `ntdll` internals, CRT +helpers like `memset` - fall back to the original four-register heuristic unchanged. + +To regenerate the index (only needed after bumping the `win32json` submodule): + +```powershell +python tools\build-win32-index.py +``` + # Prerequisites - Windows - v145 for Microsoft C++ Build Tools @@ -37,11 +73,18 @@ which has the potential to significantly increase quick wins during malware anal - Python 3.10+ (for CAPA) # Building ttd-capa +Clone with submodules (`git clone --recursive`, or `git submodule update --init` in an existing clone) - the +`win32json` submodule supplies the API metadata. + 1. Open `ttd/ttdcapa-extract.sln` in Visual Studio 2. Ensure that the required nuget packages (`Microsoft.TimeTravelDebugging.Apis` and `nlohmann.json`) are installed 3. Set the build mode to `x64` and `Release` 4. Navigate to `Build > Build Solution` in order to begin the build +The build copies `ttd/data/win32-index.bin` next to the executable. If it's missing, run +`python tools\build-win32-index.py` to generate it; without it the extractor still runs, but falls back to heuristic +argument capture for every call. + After the build, ttd-capa cannot run properly without Microsoft's `TTDReplay.dll` and `TTDReplayCPU.dll` being in the same directory as `ttdcapa-extract.exe`. To get those DLLs, ensure you have WinDbg instealled already. Then, run the following PowerShell command to find the DLL location on your system: @@ -85,7 +128,12 @@ python ttd-capa.py --sample sample.exe -- -vv Useful flags: - `--extractor ` - specify a path to the ttd-capa extractor executable - `--max-calls N` - cap huge traces to certain number of calls -- `--with-stack-args` - captures parameters for function calls that are on the stack (capture more than 5+ parameters for function calls) +- `--with-stack-args` - for calls with *no* Win32 metadata, also grab four stack slots past the register arguments. + Calls we have a signature for always capture their true arity, stack parameters included, so this only affects the + heuristic fallback path +- `--win32-index ` - use a specific `win32-index.bin` instead of the one next to the extractor +- `--no-metadata` - disable metadata-driven decoding and use the original four-register heuristic everywhere +- `--max-buffer N` - bytes to keep from any one captured buffer (default 256) - `--keep-json` - keep the generated JSON report after the Python script runs ## Manual @@ -129,8 +177,14 @@ python ttd-timeline.py --calls # Limitations - Only x64 traces are supported at the moment (no x86 or ARM) -- Argument captures are heuristic, so errors may occur - Only the functions directly exported by loaded modules are logged in the JSON report +- Argument decoding is exact only for APIs covered by the Win32 metadata (public SDK surface). `ntdll` internals, + undocumented APIs, and CRT helpers fall back to the four-register heuristic, so their arguments may still be wrong +- COM interface methods are not resolved; the metadata has no vtable indices, and the call target is a vtable slot + rather than a named export +- Variadic functions (`printf`-style) have no statically knowable argument count, so only their fixed parameters are + decoded +- Structure parameters are recorded as pointers; fields are not expanded # Verifying the backend in isolation `tests/test_ttd_extractor.py` loads a report and dumps every feature per scope - helpful to confirm expected `API`/`Number`/`String` features: diff --git a/tools/build-win32-index.py b/tools/build-win32-index.py new file mode 100644 index 0000000..53c2cb9 --- /dev/null +++ b/tools/build-win32-index.py @@ -0,0 +1,641 @@ +""" +Flatten the win32json Win32 API metadata into a compact binary index that +ttdcapa-extract loads at startup to decode call parameters. + +The metadata (https://github.com/marlersoft/win32json, vendored as the `win32json` +submodule) is ~66 MB of JSON across ~300 files. Parsing that at debug time would be +absurd, so we pre-bake it once: resolve the type graph, classify every parameter +into an x64 ABI slot plus a small "decode kind" the C++ side can switch on, and +write a single mmap-friendly blob. + + python tools/build-win32-index.py [win32json/api] [-o ttd/data/win32-index.bin] + +Regenerate whenever the win32json submodule is bumped. See +WIN32JSON-TTD-INTEGRATION-NOTES.md for what the metadata does and does not give us +(short version: types and semantics yes, ABI classification is ours -- that's the +`SLOT_*` / `classify_param` logic below). +""" +import io +import os +import sys +import json +import glob +import struct +import argparse +from pathlib import Path + +MAGIC = b"W32IDX01" +FORMAT_VERSION = 1 + +# --- decode kinds; keep in sync with ArgKind in ttd/src/win32meta.hpp ---------- +K_UNKNOWN = 0 +K_INTEGER = 1 +K_BOOL = 2 +K_HANDLE = 3 +K_ENUM = 4 +K_FLOAT = 5 +K_DOUBLE = 6 +K_ANSI_STRING = 7 +K_WIDE_STRING = 8 +K_ANSI_BUFFER = 9 +K_WIDE_BUFFER = 10 +K_BYTE_BUFFER = 11 +K_PTR_TO_INT = 12 +K_STRUCT_PTR = 13 +K_FUNC_PTR = 14 +K_GUID = 15 +K_POINTER = 16 +K_PTR_TO_ANSI_STRING = 17 +K_PTR_TO_WIDE_STRING = 18 + +# --- parameter attribute bits; keep in sync with ParamAttr in win32meta.hpp --- +A_IN = 0x01 +A_OUT = 0x02 +A_OPTIONAL = 0x04 +A_CONST = 0x08 +A_RESERVED = 0x10 +A_NOT_NUL_TERM = 0x20 +A_NULNUL_TERM = 0x40 +A_COM_OUT_PTR = 0x80 + +# --- how a buffer's length is determined; keep in sync with AuxKind ----------- +AUX_NONE = 0 +AUX_BYTES_FROM_PARAM = 1 +AUX_COUNT_FROM_PARAM = 2 +AUX_COUNT_CONST = 3 + +# --- function flags; keep in sync with FuncFlag ------------------------------ +F_HIDDEN_RET_PTR = 0x01 +F_UNSUPPORTED = 0x02 +F_SET_LAST_ERROR = 0x04 + +NO_ENUM = 0xFFFFFFFF + +# x64 sizes for the metadata's primitive names +NATIVE_SIZE = { + "Byte": 1, "SByte": 1, "Boolean": 1, + "Int16": 2, "UInt16": 2, "Char": 2, + "Int32": 4, "UInt32": 4, "Single": 4, + "Int64": 8, "UInt64": 8, "Double": 8, "IntPtr": 8, "UIntPtr": 8, + "Guid": 16, + "Void": 0, +} +NATIVE_INT = { + "Byte", "SByte", "Boolean", "Int16", "UInt16", "Char", + "Int32", "UInt32", "Int64", "UInt64", "IntPtr", "UIntPtr", +} + +# NativeTypedefs that are really strings, not opaque handles. BSTR carries a +# FreeFunc (SysFreeString) so the handle heuristic below would otherwise claim it. +STRING_TYPEDEFS = {"PSTR": K_ANSI_STRING, "PWSTR": K_WIDE_STRING, "BSTR": K_WIDE_STRING} + +MAX_PARAMS = 255 # param_count is a u8; nothing real comes close (max observed: 18) + + +class StringTable: + """Deduplicating NUL-terminated string pool. Offset 0 is always "".""" + + def __init__(self): + self.buf = bytearray(b"\x00") + self.offsets = {"": 0} + + def add(self, s): + if s is None: + s = "" + off = self.offsets.get(s) + if off is None: + off = len(self.buf) + self.buf += s.encode("utf-8") + b"\x00" + self.offsets[s] = off + return off + + +class Metadata: + """The win32json type graph, plus the resolution helpers built on top of it.""" + + def __init__(self, api_dir): + self.types = {} # (api, name) -> type dict + self.functions = [] # (api, function dict) + self.unicode_aliases = set() + files = sorted(glob.glob(os.path.join(api_dir, "*.json"))) + if not files: + sys.exit(f"no .json files under {api_dir}; is the win32json submodule checked out?") + for path in files: + api = os.path.basename(path)[:-5] + with open(path, encoding="utf-8") as f: + doc = json.load(f) + for t in doc.get("Types", []): + # first definition wins; arch-specific duplicates are handled by + # SupportedArchitecture, which we resolve to x64 at index time + self.types.setdefault((api, t["Name"]), t) + for fn in doc.get("Functions", []): + self.functions.append((api, fn)) + self.unicode_aliases.update(doc.get("UnicodeAliases", [])) + self._size_cache = {} + + def lookup(self, ref): + return self.types.get((ref.get("Api"), ref.get("Name"))) + + def resolve(self, t, depth=0): + """Follow ApiRef -> NativeTypedef chains to a terminal type node. + + Returns (node, typedef) where `typedef` is the last NativeTypedef we passed + through (or None). Callers need the typedef to spot HANDLE/PSTR, which are + only distinguishable by name -- their definitions are plain IntPtr/Byte*. + """ + seen = set() + typedef = None + while depth < 16: + depth += 1 + if t.get("Kind") != "ApiRef": + return t, typedef + key = (t.get("Api"), t.get("Name")) + if key in seen: + return t, typedef + seen.add(key) + target = self.lookup(t) + if target is None: + return t, typedef + if target.get("Kind") == "NativeTypedef": + typedef = target + t = target["Def"] + continue + return target, typedef + return t, typedef + + # --- sizeof, needed only to classify by-value aggregates ------------------ + + def sizeof(self, t, depth=0): + """x64 size in bytes, or None if it can't be determined. + + Only aggregates passed/returned by value need this (236 params and 18 + return types across the whole surface), so a None here costs us one + function, not correctness everywhere. + """ + if depth > 24: + return None + kind = t.get("Kind") + if kind == "Native": + return NATIVE_SIZE.get(t.get("Name")) + if kind in ("PointerTo", "LPArray", "FunctionPointer"): + return 8 + if kind == "Array": + child = self.sizeof(t["Child"], depth + 1) + count = (t.get("Shape") or {}).get("Size") + if child is None or not count: + return None + return child * count + if kind == "ApiRef": + node, _ = self.resolve(t) + if node.get("Kind") == "ApiRef": + return None # unresolvable reference + return self.sizeof(node, depth + 1) + if kind == "Com": + return 8 + if kind == "Enum": + return self.enum_width(t) + if kind in ("Struct", "Union"): + return self._sizeof_record(t, depth) + return None + + def _sizeof_record(self, t, depth): + key = id(t) + cached = self._size_cache.get(key) + if cached is not None: + return cached[0] + self._size_cache[key] = (None,) # cycle guard + declared = t.get("Size") or 0 + if declared: + self._size_cache[key] = (declared,) + return declared + pack = t.get("PackingSize") or 0 + offset = 0 + max_align = 1 + for field in t.get("Fields", []): + fsize = self.sizeof(field["Type"], depth + 1) + if fsize is None: + self._size_cache[key] = (None,) + return None + align = min(fsize if fsize in (1, 2, 4, 8, 16) else 8, pack) if pack else fsize + align = max(1, min(align if align in (1, 2, 4, 8, 16) else 8, 8)) + max_align = max(max_align, align) + if t["Kind"] == "Union": + offset = max(offset, fsize) + else: + offset = (offset + align - 1) // align * align + fsize + size = (offset + max_align - 1) // max_align * max_align if offset else 0 + self._size_cache[key] = (size,) + return size + + def enum_width(self, enum_type): + node = enum_type + if node.get("Kind") == "ApiRef": + node, _ = self.resolve(node) + integer_base = node.get("IntegerBase") + if integer_base: + return NATIVE_SIZE.get(integer_base, 4) + return 4 + + +def attr_bits(attrs): + """Fold a param's Attrs list into a bitmask plus its MemorySize source, if any.""" + bits = 0 + bytes_param = None + for a in attrs: + if isinstance(a, dict): + if a.get("Kind") == "MemorySize": + bytes_param = a.get("BytesParamIndex") + continue + bits |= { + "In": A_IN, "Out": A_OUT, "Optional": A_OPTIONAL, "Const": A_CONST, + "Reserved": A_RESERVED, "NotNullTerminated": A_NOT_NUL_TERM, + "NullNullTerminated": A_NULNUL_TERM, "ComOutPtr": A_COM_OUT_PTR, + }.get(a, 0) + return bits, bytes_param + + +class Param: + __slots__ = ("name", "type_name", "kind", "attrs", "slot", + "aux_kind", "aux_value", "enum_idx", "pointee_size") + + def __init__(self): + self.name = "" + self.type_name = "" + self.kind = K_UNKNOWN + self.attrs = 0 + self.slot = 0 + self.aux_kind = AUX_NONE + self.aux_value = 0 + self.enum_idx = NO_ENUM + self.pointee_size = 0 + + +def type_display_name(t): + """A short human-readable name for the report/timeline, e.g. "PWSTR", "void*".""" + kind = t.get("Kind") + if kind == "ApiRef": + return t.get("Name", "") + if kind == "Native": + return t.get("Name", "") + if kind == "PointerTo": + return type_display_name(t["Child"]) + "*" + if kind == "LPArray": + return type_display_name(t["Child"]) + "[]" + if kind == "FunctionPointer": + return "fnptr" + return kind or "" + + +def classify_pointee(md, child): + """Classify what a pointer points at -> (kind, pointee_size). + + Split out because PointerTo and LPArray share it. + """ + node, typedef = md.resolve(child) + if typedef is not None and typedef["Name"] in STRING_TYPEDEFS: + # e.g. PWSTR* -- an out-parameter that receives an allocated string + return ({K_ANSI_STRING: K_PTR_TO_ANSI_STRING, + K_WIDE_STRING: K_PTR_TO_WIDE_STRING}[STRING_TYPEDEFS[typedef["Name"]]], 8) + + kind = node.get("Kind") + if kind == "Native": + name = node.get("Name") + if name in ("Byte", "SByte"): + return K_ANSI_BUFFER, 1 + if name == "Char": + return K_WIDE_BUFFER, 2 + if name == "Void": + return K_POINTER, 0 + if name == "Guid": + return K_GUID, 16 + if name in NATIVE_INT: + return K_PTR_TO_INT, NATIVE_SIZE[name] + if name in ("Single", "Double"): + return K_PTR_TO_INT, NATIVE_SIZE[name] + return K_POINTER, 0 + if kind == "Enum": + return K_PTR_TO_INT, md.enum_width(node) + if kind in ("Struct", "Union", "Com"): + return K_STRUCT_PTR, md.sizeof(node) or 0 + if kind == "FunctionPointer": + return K_FUNC_PTR, 8 + if kind in ("PointerTo", "LPArray"): + return K_PTR_TO_INT, 8 # void** / T** -- at least surface the inner pointer + return K_POINTER, 0 + + +def classify_param(md, p, enum_ids): + """Map one metadata parameter onto a decode kind + buffer-length source. + + Returns a Param with everything except `slot` filled in (slot assignment needs + whole-signature context and happens in build_function). + """ + out = Param() + out.name = p.get("Name") or "" + t = p["Type"] + out.type_name = type_display_name(t) + out.attrs, bytes_param = attr_bits(p.get("Attrs") or []) + + kind = t.get("Kind") + + if kind == "LPArray": + out.kind, out.pointee_size = classify_pointee(md, t["Child"]) + if out.kind == K_POINTER: + out.kind = K_BYTE_BUFFER + count_param = t.get("CountParamIndex", -1) + count_const = t.get("CountConst", -1) + if count_param is not None and count_param >= 0: + out.aux_kind, out.aux_value = AUX_COUNT_FROM_PARAM, count_param + elif count_const is not None and count_const >= 0: + out.aux_kind, out.aux_value = AUX_COUNT_CONST, count_const + + elif kind == "PointerTo": + out.kind, out.pointee_size = classify_pointee(md, t["Child"]) + + elif kind == "ApiRef": + node, typedef = md.resolve(t) + if typedef is not None and typedef["Name"] in STRING_TYPEDEFS: + out.kind = STRING_TYPEDEFS[typedef["Name"]] + out.pointee_size = 1 if out.kind == K_ANSI_STRING else 2 + elif typedef is not None and is_handle_typedef(typedef): + # an opaque kernel/GDI/etc handle: pointer-sized, never dereference it + out.kind, out.pointee_size = K_HANDLE, 8 + else: + nkind = node.get("Kind") + if nkind == "Enum": + out.kind = K_ENUM + out.pointee_size = md.enum_width(node) + enum_key = (node.get("__api__"), node["Name"]) + out.enum_idx = enum_ids.get(enum_key, NO_ENUM) + elif nkind == "Native": + out.kind, out.pointee_size = classify_native(node) + elif nkind in ("Struct", "Union"): + size = md.sizeof(node) + if size in (1, 2, 4, 8): + out.kind, out.pointee_size = K_INTEGER, size + elif size is None: + out.kind = K_UNKNOWN + else: + # x64: aggregates that aren't 1/2/4/8 bytes go by hidden pointer + out.kind, out.pointee_size = K_STRUCT_PTR, size + elif nkind == "Com": + out.kind, out.pointee_size = K_POINTER, 8 + elif nkind == "FunctionPointer": + out.kind, out.pointee_size = K_FUNC_PTR, 8 + elif nkind == "PointerTo": + out.kind, out.pointee_size = classify_pointee(md, node["Child"]) + else: + out.kind = K_UNKNOWN + + elif kind == "Native": + out.kind, out.pointee_size = classify_native(t) + if out.kind == K_GUID: + out.kind = K_STRUCT_PTR # 16 bytes by value -> hidden pointer on x64 + + elif kind == "FunctionPointer": + out.kind, out.pointee_size = K_FUNC_PTR, 8 + + else: + out.kind = K_UNKNOWN + + # a MemorySize attribute always wins: it says this really is a sized buffer + if bytes_param is not None and bytes_param >= 0: + out.aux_kind, out.aux_value = AUX_BYTES_FROM_PARAM, bytes_param + if out.kind in (K_POINTER, K_UNKNOWN, K_STRUCT_PTR): + out.kind = K_BYTE_BUFFER + out.pointee_size = 1 + + return out + + +def is_handle_typedef(typedef): + """Is this NativeTypedef an opaque handle we must never dereference? + + RAIIFree/InvalidHandleValue mark most of them (HANDLE, HKEY, ...). The rest are + pointer-sized H-prefixed typedefs with no lifetime metadata (HWND, HDESK, ...); + treating those as integers would be harmless but renders worse. + """ + if typedef.get("FreeFunc") or typedef.get("InvalidHandleValue") is not None: + return True + name = typedef["Name"] + return ( + name.startswith("H") + and typedef["Def"].get("Kind") == "Native" + and typedef["Def"].get("Name") in ("IntPtr", "UIntPtr") + ) + + +def classify_native(t): + name = t.get("Name") + if name == "Single": + return K_FLOAT, 4 + if name == "Double": + return K_DOUBLE, 8 + if name == "Boolean": + return K_BOOL, 1 + if name == "Guid": + return K_GUID, 16 + if name == "Void": + return K_UNKNOWN, 0 + if name in NATIVE_INT: + return K_INTEGER, NATIVE_SIZE[name] + return K_UNKNOWN, 0 + + +def wants_x64(arches): + """Architectures==[] means arch-neutral; otherwise it must list X64.""" + return not arches or "X64" in arches + + +def build_function(md, api, fn, enum_ids): + """Produce (params, flags) for one function, or None if it isn't x64-relevant.""" + if not wants_x64(fn.get("Architectures") or []): + return None + + flags = F_SET_LAST_ERROR if fn.get("SetLastError") else 0 + + # A function returning an aggregate that isn't 1/2/4/8 bytes takes a hidden + # pointer in RCX, pushing every real parameter one slot to the right. + ret = fn.get("ReturnType") or {} + ret_node, _ = md.resolve(ret) if ret.get("Kind") == "ApiRef" else (ret, None) + base_slot = 0 + if ret_node.get("Kind") in ("Struct", "Union") or ( + ret_node.get("Kind") == "Native" and ret_node.get("Name") == "Guid" + ): + size = md.sizeof(ret_node) + if size is None: + flags |= F_UNSUPPORTED + elif size not in (1, 2, 4, 8): + flags |= F_HIDDEN_RET_PTR + base_slot = 1 + + raw_params = fn.get("Params") or [] + if len(raw_params) + base_slot > MAX_PARAMS: + return None + + params = [] + for i, p in enumerate(raw_params): + cp = classify_param(md, p, enum_ids) + cp.slot = base_slot + i + if cp.kind == K_UNKNOWN: + flags |= F_UNSUPPORTED + params.append(cp) + + # aux_value indexes into the *parameter* list; the C++ side reads captured + # values by parameter index too, so no slot translation is needed. Drop + # references that point outside the list rather than trusting them at runtime. + for cp in params: + if cp.aux_kind in (AUX_BYTES_FROM_PARAM, AUX_COUNT_FROM_PARAM): + if not (0 <= cp.aux_value < len(params)): + cp.aux_kind, cp.aux_value = AUX_NONE, 0 + + return params, flags + + +def collect_enums(md): + """Index only the enums a parameter can actually reference (950 of 7005).""" + referenced = {} + + def visit(t, depth=0): + if depth > 16: + return + kind = t.get("Kind") + if kind == "ApiRef": + node, _ = md.resolve(t) + if node.get("Kind") == "Enum": + referenced.setdefault((t.get("Api"), t.get("Name")), node) + elif kind in ("PointerTo", "LPArray"): + visit(t["Child"], depth + 1) + + for api, fn in md.functions: + for p in fn.get("Params") or []: + visit(p["Type"]) + + enums = [] + enum_ids = {} + for (eapi, ename), node in sorted(referenced.items()): + node = dict(node) + node["__api__"] = eapi + enum_ids[(eapi, ename)] = len(enums) + enums.append((eapi, ename, node)) + return enums, enum_ids + + +def resolve_enum_index(md, t, enum_ids, depth=0): + """The enum table index a param type refers to, or NO_ENUM.""" + if depth > 16: + return NO_ENUM + kind = t.get("Kind") + if kind == "ApiRef": + node, _ = md.resolve(t) + if node.get("Kind") == "Enum": + return enum_ids.get((t.get("Api"), t.get("Name")), NO_ENUM) + elif kind in ("PointerTo", "LPArray"): + return resolve_enum_index(md, t["Child"], enum_ids, depth + 1) + return NO_ENUM + + +def build(api_dir, out_path): + print(f"[+] loading {api_dir} ...") + md = Metadata(api_dir) + print(f"[+] {len(md.functions)} functions, {len(md.types)} types") + + enums, enum_ids = collect_enums(md) + print(f"[+] {len(enums)} referenced enums") + + strtab = StringTable() + + # enum tables + enum_recs = [] + enumval_recs = [] + for eapi, ename, node in enums: + values = node.get("Values") or [] + val_off = len(enumval_recs) + for v in values: + enumval_recs.append((strtab.add(v["Name"]), int(v["Value"]))) + enum_recs.append(( + strtab.add(ename), val_off, len(values), + 1 if node.get("Flags") else 0, md.enum_width(node), + )) + + # functions, keyed by bare export name (module is a weak hint only -- API sets + # mean kernel32!CreateFileW surfaces as KERNELBASE!CreateFileW in a trace) + func_recs = [] + param_recs = [] + seen_names = {} + skipped_arch = 0 + unsupported = 0 + collisions = 0 + + for api, fn in sorted(md.functions, key=lambda x: x[1]["Name"]): + name = fn["Name"] + built = build_function(md, api, fn, enum_ids) + if built is None: + skipped_arch += 1 + continue + params, flags = built + + if name in seen_names: + prior = seen_names[name] + if prior != tuple((p.kind, p.slot) for p in params): + collisions += 1 + continue + seen_names[name] = tuple((p.kind, p.slot) for p in params) + + for i, p in enumerate(params): + enum_idx = resolve_enum_index(md, fn["Params"][i]["Type"], enum_ids) + param_recs.append(( + strtab.add(p.name), strtab.add(p.type_name), p.kind, p.attrs, + p.slot, p.aux_kind, p.aux_value, enum_idx, p.pointee_size, + )) + if flags & F_UNSUPPORTED: + unsupported += 1 + func_recs.append(( + strtab.add(name), strtab.add(fn.get("DllImport") or ""), + len(param_recs) - len(params), len(params), flags, + )) + + print(f"[+] indexed {len(func_recs)} functions " + f"({skipped_arch} not x64, {unsupported} flagged UNSUPPORTED, " + f"{collisions} name collisions with differing shapes)") + print(f"[+] {len(param_recs)} params, {len(enumval_recs)} enum values, " + f"{len(strtab.buf)} bytes of strings") + + blob = io.BytesIO() + blob.write(MAGIC) + blob.write(struct.pack( + " int: parser.add_argument("--sample", help="optional on-disk sample for accurate hashes") parser.add_argument("--extractor", help="path to ttdcapa-extract.exe") parser.add_argument("--max-calls", type=int, help="cap recorded API calls (for huge traces)") - parser.add_argument("--with-stack-args", action="store_true", help="also capture stack args 5+") + parser.add_argument( + "--with-stack-args", + action="store_true", + help="for calls with no Win32 metadata, also grab four stack slots past the " + "register args (calls we have a signature for always capture their true arity)", + ) + parser.add_argument("--win32-index", help="path to win32-index.bin (default: next to the extractor)") + parser.add_argument( + "--no-metadata", + action="store_true", + help="disable metadata-driven argument decoding entirely", + ) + parser.add_argument("--max-buffer", type=int, help="bytes kept from any one captured buffer (default 256)") parser.add_argument("--keep-json", action="store_true", help="keep the intermediate TTD report") parser.add_argument("--python", default=sys.executable, help="python interpreter to run capa") args, capa_extra = parser.parse_known_args(argv) @@ -71,9 +83,10 @@ def main(argv: list[str]) -> int: trace = Path(args.trace) if not trace.is_file(): sys.exit(f"trace not found: {trace}") - rules = Path(args.rules) + # absolute, because capa is invoked with cwd set to the report's directory + rules = Path(args.rules).resolve() if not rules.exists(): - sys.exit(f"rules path not found: {rules}") + sys.exit(f"rules path not found: {args.rules}") extractor = find_extractor(args.extractor) @@ -89,6 +102,12 @@ def main(argv: list[str]) -> int: extract_cmd += ["--max-calls", str(args.max_calls)] if args.with_stack_args: extract_cmd += ["--with-stack-args"] + if args.win32_index: + extract_cmd += ["--win32-index", args.win32_index] + if args.no_metadata: + extract_cmd += ["--no-metadata"] + if args.max_buffer: + extract_cmd += ["--max-buffer", str(args.max_buffer)] print(f"[ttd-capa] extracting: {' '.join(extract_cmd)}", file=sys.stderr) rc = subprocess.call(extract_cmd) diff --git a/ttd-timeline.py b/ttd-timeline.py index 9eb6c1c..c172710 100644 --- a/ttd-timeline.py +++ b/ttd-timeline.py @@ -28,6 +28,20 @@ from typing import Optional HERE = Path(__file__).resolve().parent +# Recovered guest strings are arbitrary text -- non-Latin paths, HTTP bodies, the +# occasional lone surrogate from a half-written UTF-16 buffer. When stdout is a +# console Python uses UTF-8, but when it is redirected to a file it falls back to +# the locale encoding (cp1252 here), which cannot encode any of that and aborts the +# whole run. Pin UTF-8 and degrade unencodable characters instead of dying. +for _stream in (sys.stdout, sys.stderr): + if hasattr(_stream, "reconfigure"): + _stream.reconfigure(encoding="utf-8", errors="replace") + +# Python puts this script's directory first on sys.path, and the capa fork lives in +# a `capa/` subdirectory of it -- which shadows the installed `capa` package with a +# namespace package that has no submodules. Drop it; nothing here is imported locally. +sys.path[:] = [p for p in sys.path if p and Path(p).resolve() != HERE] + from capa.features.address import ProcessAddress, ThreadAddress, DynamicCallAddress # noqa: E402 from capa.features.extractors.ttd.models import TtdCall # noqa: E402 from capa.features.extractors.ttd.extractor import TtdExtractor # noqa: E402 @@ -66,9 +80,33 @@ def fmt_arg(a) -> str: return repr(a) +def fmt_param(p) -> str: + """One metadata-decoded parameter as `name=value`. + + Prefers the most informative rendering the extractor managed: decoded text, + then symbolic flag names, then a dereferenced pointee, then the raw value. + """ + if p.str_ is not None: + value = repr(p.str_) + elif p.flags: + value = "|".join(p.flags) + elif p.float_ is not None: + value = repr(p.float_) + else: + value = fmt_arg(p.value) + if p.deref is not None: + value += f"->{fmt_arg(p.deref)}" + if p.at_return: + value += "@ret" + return f"{p.name}={value}" if p.name else value + + def format_call(call: TtdCall) -> str: api = f"{call.module}.{call.api}" if call.module else call.api - args = ", ".join(fmt_arg(a) for a in call.args) + if call.params: + args = ", ".join(fmt_param(p) for p in call.params) + else: + args = ", ".join(fmt_arg(a) for a in call.args) ret = "" if call.ret is None else f" -> 0x{call.ret & 0xFFFFFFFFFFFFFFFF:x}" return f"{api}({args}){ret}" diff --git a/ttd/data/win32-index.bin b/ttd/data/win32-index.bin new file mode 100644 index 0000000..9df5b24 Binary files /dev/null and b/ttd/data/win32-index.bin differ diff --git a/ttd/src/abi_x64.cpp b/ttd/src/abi_x64.cpp new file mode 100644 index 0000000..288237e --- /dev/null +++ b/ttd/src/abi_x64.cpp @@ -0,0 +1,422 @@ +#include "abi_x64.hpp" + +#include +#include + +using ttdcapa::win32meta::ArgKind; +using ttdcapa::win32meta::AuxKind; + +namespace ttdcapa { + namespace { + + // A dereference is only worth attempting above the first 64 KiB; the null + // page and its neighbours are never mapped in user mode. + constexpr uint64_t kMinDerefAddr = 0x10000; + + // Guards against a mis-typed count parameter turning into a huge read. + constexpr uint64_t kMaxCountElements = 1u << 20; + + bool readGuest(TTD::Replay::IThreadView const* thread, uint64_t addr, void* dst, size_t size) { + if (addr < kMinDerefAddr || size == 0) { + return false; + } + auto result = thread->QueryMemoryBuffer(TTD::GuestAddress{ addr }, TTD::BufferView{ dst, size }); + return result.Memory.Size == size; + } + + // The value in `slot`, honouring the shared integer/SSE slot numbering. + uint64_t fetchSlot(const AMD64_CONTEXT& ctx, TTD::Replay::IThreadView const* thread, + uint8_t slot, bool isFloat, bool& ok) { + ok = true; + if (slot < 4) { + if (isFloat) { + const M128BIT* xmm[4] = { &ctx.Xmm0, &ctx.Xmm1, &ctx.Xmm2, &ctx.Xmm3 }; + return xmm[slot]->Low; + } + const uint64_t gpr[4] = { ctx.Rcx, ctx.Rdx, ctx.R8, ctx.R9 }; + return gpr[slot]; + } + // Above the shadow space the caller reserved for RCX/RDX/R8/R9. + uint64_t addr = ctx.Rsp + 0x28 + static_cast(slot - 4) * 8; + uint64_t v = 0; + ok = readGuest(thread, addr, &v, sizeof(v)); + return v; + } + + double floatValue(const win32meta::ParamSig& p, uint64_t bits) { + if (p.kind == ArgKind::Float) { + float f = 0.0f; + uint32_t lo = static_cast(bits); + std::memcpy(&f, &lo, sizeof(f)); + return static_cast(f); + } + double d = 0.0; + std::memcpy(&d, &bits, sizeof(d)); + return d; + } + + // Widen a pointee of `size` bytes to 64 bits. + uint64_t narrowRead(const uint8_t* raw, uint16_t size) { + uint64_t v = 0; + std::memcpy(&v, raw, size > 8 ? 8 : size); + return v; + } + + bool derefScalar(TTD::Replay::IThreadView const* thread, uint64_t ptr, uint16_t size, uint64_t& out) { + uint16_t width = size == 0 ? 8 : (size > 8 ? 8 : size); + uint8_t buf[8] = {}; + if (!readGuest(thread, ptr, buf, width)) { + return false; + } + out = narrowRead(buf, width); + return true; + } + + std::string formatGuid(const uint8_t* b) { + char buf[40]; + std::snprintf(buf, sizeof(buf), + "{%08lX-%04X-%04X-%02X%02X-%02X%02X%02X%02X%02X%02X}", + static_cast(narrowRead(b, 4)), + static_cast(narrowRead(b + 4, 2)), + static_cast(narrowRead(b + 6, 2)), + b[8], b[9], b[10], b[11], b[12], b[13], b[14], b[15]); + return buf; + } + + bool isBufferKind(ArgKind kind) { + return kind == ArgKind::AnsiBuffer || kind == ArgKind::WideBuffer || kind == ArgKind::ByteBuffer; + } + + // How many bytes a counted buffer parameter spans, or 0 when we can't tell. + // `args` supplies the sibling parameter the count lives in -- after a return + // pass those may themselves have been filled in, which is exactly what makes + // ReadFile's lpBuffer renderable at its *actual* length. + uint64_t resolveByteCount(const win32meta::ParamSig& p, const std::vector& args) { + uint64_t count = 0; + switch (p.auxKind) { + case AuxKind::CountConst: + count = static_cast(p.auxValue); + break; + case AuxKind::BytesFromParam: + case AuxKind::CountFromParam: { + if (p.auxValue < 0 || static_cast(p.auxValue) >= args.size()) { + return 0; + } + const DecodedArg& src = args[static_cast(p.auxValue)]; + count = src.has_deref ? src.deref : src.raw; + break; + } + case AuxKind::None: + default: + return 0; + } + if (count == 0 || count > kMaxCountElements) { + return 0; + } + if (p.auxKind == AuxKind::BytesFromParam) { + return count; + } + uint16_t elem = p.pointeeSize ? p.pointeeSize : 1; + return count * elem; + } + + // Length of `buf` up to and including its first `charWidth`-wide NUL. A + // character buffer's count parameter is the caller's *capacity*, so without + // this the report would carry a few hundred bytes of unrelated stack memory + // after every out-string. + size_t terminatorEnd(const std::vector& buf, size_t charWidth) { + for (size_t i = 0; i + charWidth <= buf.size(); i += charWidth) { + bool nul = true; + for (size_t k = 0; k < charWidth; ++k) { + if (buf[i + k] != 0) { + nul = false; + break; + } + } + if (nul) { + return i + charWidth; + } + } + return buf.size(); + } + + // Read a counted buffer into `arg`, capped at opt.max_buffer. Character + // buffers additionally get a textual rendering, since that's what a rule or + // an analyst actually wants to see. + void captureBuffer(TTD::Replay::IThreadView const* thread, const DecodeOptions& opt, + ArgKind kind, uint64_t ptr, uint64_t byteCount, DecodedArg& arg) { + if (ptr < kMinDerefAddr || byteCount == 0) { + return; + } + size_t want = static_cast(byteCount < opt.max_buffer ? byteCount : opt.max_buffer); + std::vector buf(want); + auto result = thread->QueryMemoryBuffer(TTD::GuestAddress{ ptr }, TTD::BufferView{ buf.data(), want }); + size_t got = result.Memory.Size; + if (got == 0) { + return; + } + buf.resize(got); + + if (kind == ArgKind::AnsiBuffer) { + if (auto s = readAnsiString(thread, ptr, got)) { + arg.str = std::move(*s); + arg.has_str = !arg.str.empty(); + } + buf.resize(terminatorEnd(buf, 1)); + } else if (kind == ArgKind::WideBuffer) { + if (auto s = readWideString(thread, ptr, got / sizeof(wchar_t))) { + arg.str = std::move(*s); + arg.has_str = !arg.str.empty(); + } + buf.resize(terminatorEnd(buf, 2)); + } + arg.bytes = std::move(buf); + } + + // Everything that needs the callee to have run. Shared by the entry pass + // (for [In] parameters, which are already valid) and the return pass. + void dereference(TTD::Replay::IThreadView const* thread, const DecodeOptions& opt, + ArgKind kind, uint64_t ptr, uint16_t pointeeSize, + uint64_t byteCount, DecodedArg& arg) { + switch (kind) { + case ArgKind::AnsiString: + if (auto s = readAnsiString(thread, ptr, opt.max_string)) { + arg.str = std::move(*s); + arg.has_str = true; + } + break; + case ArgKind::WideString: + if (auto s = readWideString(thread, ptr, opt.max_string)) { + arg.str = std::move(*s); + arg.has_str = true; + } + break; + case ArgKind::PtrToInt: { + uint64_t v = 0; + if (derefScalar(thread, ptr, pointeeSize, v)) { + arg.deref = v; + arg.has_deref = true; + } + break; + } + case ArgKind::PtrToAnsiString: + case ArgKind::PtrToWideString: { + uint64_t inner = 0; + if (!derefScalar(thread, ptr, 8, inner)) { + break; + } + arg.deref = inner; + arg.has_deref = true; + auto s = (kind == ArgKind::PtrToAnsiString) + ? readAnsiString(thread, inner, opt.max_string) + : readWideString(thread, inner, opt.max_string); + if (s) { + arg.str = std::move(*s); + arg.has_str = true; + } + break; + } + case ArgKind::Guid: { + uint8_t g[16] = {}; + if (readGuest(thread, ptr, g, sizeof(g))) { + arg.str = formatGuid(g); + arg.has_str = true; + } + break; + } + case ArgKind::AnsiBuffer: + case ArgKind::WideBuffer: + case ArgKind::ByteBuffer: + captureBuffer(thread, opt, kind, ptr, byteCount, arg); + break; + default: + break; + } + } + + // Some parameters are pointers whose pointee the metadata can't pin down: + // opaque void*, and the raw UInt16*/UIntPtr* that RPC uses for RPC_WSTR. + // For those the old guess-if-it-looks-like-text heuristic is still the best + // information available -- and unlike before, we now only apply it to values + // we know really are parameters and really are pointers. + bool mayHoldUntypedString(ArgKind kind) { + return kind == ArgKind::Pointer || kind == ArgKind::Unknown || kind == ArgKind::PtrToInt; + } + + void tryUntypedString(TTD::Replay::IThreadView const* thread, DecodedArg& arg) { + if (arg.has_str || !mayHoldUntypedString(arg.kind)) { + return; + } + if (auto s = tryReadString(thread, arg.raw)) { + arg.str = std::move(*s); + arg.has_str = true; + return; + } + // A T** out-parameter: the string lives one more hop away. + if (arg.has_deref) { + if (auto s = tryReadString(thread, arg.deref)) { + arg.str = std::move(*s); + arg.has_str = true; + } + } + } + + bool needsDeref(ArgKind kind) { + switch (kind) { + case ArgKind::AnsiString: + case ArgKind::WideString: + case ArgKind::PtrToInt: + case ArgKind::PtrToAnsiString: + case ArgKind::PtrToWideString: + case ArgKind::Guid: + case ArgKind::AnsiBuffer: + case ArgKind::WideBuffer: + case ArgKind::ByteBuffer: + return true; + default: + return false; + } + } + + } // namespace + + void decodeArgs(const win32meta::FuncSig& sig, + const AMD64_CONTEXT& ctx, + TTD::Replay::IThreadView const* thread, + const DecodeOptions& opt, + std::vector& out, + std::vector& deferred) { + out.clear(); + out.resize(sig.paramCount); + + // Pass 1: capture every raw slot first. A buffer's length can live in a + // parameter that comes *after* it (ReadFile's lpBuffer refers forward to + // nNumberOfBytesToRead), so no dereferencing until all the scalars are in. + for (uint8_t i = 0; i < sig.paramCount; ++i) { + const win32meta::ParamSig& p = sig.params[i]; + DecodedArg& arg = out[i]; + arg.name = p.name; + arg.type = p.type; + arg.kind = p.kind; + arg.enum_index = p.enumIndex; + arg.is_out = p.isOut(); + + bool ok = false; + arg.raw = fetchSlot(ctx, thread, p.slot, p.isFloat(), ok); + if (!ok) { + // Stack slot we couldn't read: leave it zero rather than invent one. + arg.raw = 0; + } + if (p.isFloat()) { + arg.fval = floatValue(p, arg.raw); + arg.has_fval = true; + } + } + + // Pass 2: dereference. Scalars and handles are deliberately untouched -- + // that alone removes most of the bogus String features the old heuristic + // produced from flag values that happened to look like addresses. + for (uint8_t i = 0; i < sig.paramCount; ++i) { + const win32meta::ParamSig& p = sig.params[i]; + DecodedArg& arg = out[i]; + if (!needsDeref(p.kind)) { + tryUntypedString(thread, arg); + continue; + } + + uint64_t byteCount = isBufferKind(p.kind) ? resolveByteCount(p, out) : 0; + + // [In] contents are already valid here. [In,Out] gets read twice: once + // now for what the caller passed, then again at the return. + if (p.isIn()) { + dereference(thread, opt, p.kind, arg.raw, p.pointeeSize, byteCount, arg); + tryUntypedString(thread, arg); + } + if (p.isOut() && arg.raw >= kMinDerefAddr) { + PendingOut pending; + pending.param_index = i; + pending.kind = p.kind; + pending.ptr = arg.raw; + pending.pointee_size = p.pointeeSize; + pending.aux_kind = p.auxKind; + pending.aux_value = p.auxValue; + pending.in_cap = byteCount; + deferred.push_back(pending); + } + } + } + + void resolvePendingOuts(const std::vector& pending, + TTD::Replay::IThreadView const* thread, + const DecodeOptions& opt, + std::vector& args) { + // Scalars first: a buffer's real length is usually itself an [Out] scalar + // (ReadFile's lpNumberOfBytesRead), so it has to be resolved before the + // buffer that depends on it. + for (const PendingOut& po : pending) { + if (po.param_index >= args.size() || isBufferKind(po.kind)) { + continue; + } + DecodedArg& arg = args[po.param_index]; + DecodedArg fresh; + dereference(thread, opt, po.kind, po.ptr, po.pointee_size, 0, fresh); + if (fresh.has_deref || fresh.has_str) { + fresh.name = arg.name; + fresh.type = arg.type; + fresh.kind = arg.kind; + fresh.enum_index = arg.enum_index; + fresh.raw = arg.raw; + fresh.is_out = true; + fresh.from_return = true; + arg = std::move(fresh); + } + tryUntypedString(thread, arg); + } + + for (const PendingOut& po : pending) { + if (po.param_index >= args.size() || !isBufferKind(po.kind)) { + continue; + } + DecodedArg& arg = args[po.param_index]; + + // Prefer the length the callee reported; fall back to what the caller + // offered, and never exceed it. + win32meta::ParamSig probe; + probe.auxKind = po.aux_kind; + probe.auxValue = po.aux_value; + probe.pointeeSize = po.pointee_size; + uint64_t byteCount = resolveByteCount(probe, args); + if (byteCount == 0) { + byteCount = po.in_cap; + } else if (po.in_cap != 0 && byteCount > po.in_cap) { + byteCount = po.in_cap; + } + if (byteCount == 0) { + continue; + } + + DecodedArg fresh; + captureBuffer(thread, opt, po.kind, po.ptr, byteCount, fresh); + if (!fresh.bytes.empty() || fresh.has_str) { + arg.bytes = std::move(fresh.bytes); + arg.str = std::move(fresh.str); + arg.has_str = fresh.has_str; + arg.from_return = true; + } + } + } + + std::vector toCapaArgs(const std::vector& args) { + std::vector out; + out.reserve(args.size()); + for (const DecodedArg& a : args) { + if (a.has_str && !a.str.empty()) { + out.push_back(a.str); + } else { + out.push_back(static_cast(a.raw)); + } + } + return out; + } + +} // namespace ttdcapa diff --git a/ttd/src/abi_x64.hpp b/ttd/src/abi_x64.hpp new file mode 100644 index 0000000..65a1730 --- /dev/null +++ b/ttd/src/abi_x64.hpp @@ -0,0 +1,60 @@ +#ifndef ABI_X64_HPP +#define ABI_X64_HPP + +// The x64 calling-convention decoder. +// +// The Win32 metadata gives us types and semantics; mapping those onto registers +// and stack slots is ours to write (WIN32JSON-TTD-INTEGRATION-NOTES.md section 3). +// That's this file: given a signature and a thread's register state at a CALL, +// produce one DecodedArg per real parameter -- no more, no less. +// +// Microsoft x64 in one paragraph: the first four parameters go in RCX/RDX/R8/R9, +// or XMM0-3 if they're floating point. Crucially the slot index is *shared* between +// the two register files, so a float in position 2 lives in XMM2, not XMM0. +// Parameters five and up sit at [RSP+0x28] onwards, above the 32-byte shadow space +// the caller must reserve. Aggregates that aren't exactly 1/2/4/8 bytes are passed +// by hidden pointer, and a function returning one takes an extra hidden first +// parameter -- the indexer pre-computes both, so `slot` here is always final. + +#include +#include + +#include +#include + +#include "ttdutils.hpp" +#include "win32meta.hpp" + +namespace ttdcapa { + + struct DecodeOptions { + size_t max_buffer = 256; // bytes of any one counted buffer to keep + size_t max_string = 512; // characters of any one string to keep + }; + + // Decode every parameter of `sig` from the register state at the call. + // Dereferences that only make sense once the callee has run are appended to + // `deferred` instead; feed those to resolvePendingOuts at the return. + void decodeArgs(const win32meta::FuncSig& sig, + const AMD64_CONTEXT& ctx, + TTD::Replay::IThreadView const* thread, + const DecodeOptions& opt, + std::vector& out, + std::vector& deferred); + + // Re-read the deferred dereferences at the return position and patch them into + // `args` (which must be the vector decodeArgs filled in for this same call). + void resolvePendingOuts(const std::vector& pending, + TTD::Replay::IThreadView const* thread, + const DecodeOptions& opt, + std::vector& args); + + // Flatten decoded parameters into the int/string list capa matches rules + // against. Strings recovered from [Out] parameters are included -- they are + // genuine new evidence; symbolic enum names are not, so rules that match flags + // numerically keep working. + std::vector toCapaArgs(const std::vector& args); + +} // namespace ttdcapa + +#endif diff --git a/ttd/src/main.cpp b/ttd/src/main.cpp index 39aeefb..d3793b2 100644 --- a/ttd/src/main.cpp +++ b/ttd/src/main.cpp @@ -3,7 +3,13 @@ // (capa/features/extractors/ttd/). x64 traces only in v1. // // ttdcapa-extract [--sample ] [-o ] -// [--max-calls N] [--with-stack-args] +// [--max-calls N] [--with-stack-args] [--win32-index ] +// [--no-metadata] [--max-buffer N] +// ttdcapa-extract --dump-sig +// +// Arguments are decoded against the pre-baked Win32 metadata index whenever the +// resolved export is in it (see win32meta.hpp / abi_x64.hpp); calls we have no +// signature for fall back to the original heuristic capture. #include #define DBG_ASSERT(cond) assert(cond) @@ -30,18 +36,67 @@ #include "ttdutils.hpp" #include "utils.hpp" #include "ttd_pe_utils.hpp" +#include "win32meta.hpp" +#include "abi_x64.hpp" using namespace ttdcapa; Report g_report; +// Print one function's decoded signature and exit. Lets the metadata index and the +// ABI classification be checked without waiting on a trace replay. +static int dumpSignature(const std::string& api) { + const win32meta::FuncSig* sig = win32meta::index().lookup(api); + if (sig == nullptr) { + std::cerr << "[-] '" << api << "' is not in the metadata index\n"; + return 3; + } + std::cout << sig->name << " dll=" << sig->dll + << " params=" << static_cast(sig->paramCount) + << (sig->hiddenRetPtr() ? " [hidden-return-pointer]" : "") + << (sig->unsupported() ? " [unsupported]" : "") << "\n"; + for (uint8_t i = 0; i < sig->paramCount; ++i) { + const win32meta::ParamSig& p = sig->params[i]; + std::cout << " slot " << static_cast(p.slot) << " " << p.name + << " : " << p.type << " (" << win32meta::kindName(p.kind) << ")"; + if (p.isIn()) std::cout << " in"; + if (p.isOut()) std::cout << " out"; + if (p.attrs & win32meta::AttrOptional) std::cout << " optional"; + if (p.auxKind == win32meta::AuxKind::BytesFromParam) std::cout << " bytes=param[" << p.auxValue << "]"; + if (p.auxKind == win32meta::AuxKind::CountFromParam) std::cout << " count=param[" << p.auxValue << "]"; + if (p.auxKind == win32meta::AuxKind::CountConst) std::cout << " count=" << p.auxValue; + if (p.hasEnum()) std::cout << " enum=" << win32meta::index().enumName(p.enumIndex); + std::cout << "\n"; + } + return 0; +} + int wmain(int argc, wchar_t** argv) { Options opt; if (!parse_args(argc, argv, opt)) { - std::cerr << "Usage: ttdcapa-extract [--sample ] [-o ] [--max-calls N] [--with-stack-args]\n"; + std::cerr << "Usage: ttdcapa-extract [--sample ] [-o ]\n" + " [--max-calls N] [--with-stack-args]\n" + " [--win32-index ] [--no-metadata] [--max-buffer N]\n" + " ttdcapa-extract --dump-sig \n"; return 1; } - + + DecodeOptions decode_opt; + decode_opt.max_buffer = opt.max_buffer; + if (!opt.no_metadata) { + std::string err; + if (win32meta::loadIndex(opt.win32_index, err)) { + std::cerr << "[+] Loaded Win32 metadata for " << win32meta::index().functionCount() + << " functions\n"; + } else { + std::cerr << "[!] " << err << "; falling back to heuristic argument capture\n"; + } + } + + if (!opt.dump_sig.empty()) { + return dumpSignature(opt.dump_sig); + } + auto [engine, hr] = TTD::Replay::MakeReplayEngine(); if (hr != S_OK || !engine) { std::cerr << "[-] Failed to create replay engine: 0x" << std::hex << hr << "\n"; @@ -71,7 +126,8 @@ int wmain(int argc, wchar_t** argv) { // Get list of function VAs used for call sweep std::unordered_map> resolvedTraceModuleExports; resolvedTraceModuleExports = resolveTraceModuleExports(engine, inspection_cursor); - std::cerr << "[+] Resolved all exported module functions across execution\n"; + std::cerr << "[+] Resolved " << resolvedTraceModuleExports.size() + << " exported module functions across execution\n"; size_t thread_count = engine->GetThreadCount(); TTD::Replay::ThreadInfo const* thread_list = engine->GetThreadList(); @@ -79,9 +135,16 @@ int wmain(int argc, wchar_t** argv) { g_report.process.threads.push_back(static_cast(thread_list[i].UniqueId)); } - // per-thread stack of in-flight recorded calls: (expected return addr, call index) - std::unordered_map>> in_flight; + // per-thread stack of recorded calls we haven't seen return yet, each carrying + // the [Out] dereferences we deliberately postponed until the callee has run + struct InFlight { + uint64_t ret_addr = 0; + size_t call_index = 0; + std::vector pending; + }; + std::unordered_map> in_flight; uint64_t seq = 0; + uint64_t events_seen = 0; bool limit_hit = false; TTD::Replay::UniqueCursor sweep{ engine->NewCursor() }; @@ -90,6 +153,7 @@ int wmain(int argc, wchar_t** argv) { TTD::Replay::IThreadView const* thread) noexcept { bool is_call = (static_cast(fall_through) != 0); uint64_t utid = static_cast(thread->GetThreadInfo().UniqueId); + ++events_seen; if (is_call) { auto it = resolvedTraceModuleExports.find(static_cast(target)); @@ -120,33 +184,51 @@ int wmain(int argc, wchar_t** argv) { TTD::Replay::RegisterContext regs = thread->GetCrossPlatformContext(); auto const* ctx = reinterpret_cast(®s); - rec.args.push_back(captureCallArg(thread, ctx->Rcx)); - rec.args.push_back(captureCallArg(thread, ctx->Rdx)); - rec.args.push_back(captureCallArg(thread, ctx->R8)); - rec.args.push_back(captureCallArg(thread, ctx->R9)); - if (opt.with_stack_args) { // Capture arguments on stack based on offsets from RSP - for (int k = 0; k < 4; ++k) { - uint64_t slot = ctx->Rsp + 0x28 + static_cast(k) * 8; - uint64_t v = 0; - if (thread->QueryMemoryBuffer(TTD::GuestAddress{ slot }, TTD::BufferView{ &v, sizeof(v) }) - .Memory.Size == sizeof(v)) { - rec.args.push_back(captureCallArg(thread, v)); + std::vector pending; + win32meta::FuncSig const* sig = win32meta::index().lookup(rec.api); + if (sig != nullptr && !sig->unsupported()) { + // We know the real arity, so capture exactly that many arguments + // -- no stale RDX/R8/R9 residue masquerading as parameters. + decodeArgs(*sig, *ctx, thread, decode_opt, rec.params, pending); + rec.args = toCapaArgs(rec.params); + rec.metadata = true; + } else { + rec.args.push_back(captureCallArg(thread, ctx->Rcx)); + rec.args.push_back(captureCallArg(thread, ctx->Rdx)); + rec.args.push_back(captureCallArg(thread, ctx->R8)); + rec.args.push_back(captureCallArg(thread, ctx->R9)); + + if (opt.with_stack_args) { // Capture arguments on stack based on offsets from RSP + for (int k = 0; k < 4; ++k) { + uint64_t slot = ctx->Rsp + 0x28 + static_cast(k) * 8; + uint64_t v = 0; + if (thread->QueryMemoryBuffer(TTD::GuestAddress{ slot }, TTD::BufferView{ &v, sizeof(v) }) + .Memory.Size == sizeof(v)) { + rec.args.push_back(captureCallArg(thread, v)); + } } } } size_t idx = g_report.process.calls.size(); - in_flight[utid].push_back({ static_cast(fall_through), idx }); + in_flight[utid].push_back(InFlight{ static_cast(fall_through), idx, std::move(pending) }); g_report.process.calls.push_back(std::move(rec)); } else { // Log most recent function called as returned and capture return value auto it = in_flight.find(utid); if (it != in_flight.end() && !it->second.empty()) { - auto& top = it->second.back(); - if (top.first == static_cast(target)) { - g_report.process.calls[top.second].ret = thread->GetBasicReturnValue(); - g_report.process.calls[top.second].has_ret = true; + InFlight& top = it->second.back(); + if (top.ret_addr == static_cast(target)) { + CallRecord& call = g_report.process.calls[top.call_index]; + call.ret = thread->GetBasicReturnValue(); + call.has_ret = true; + if (!top.pending.empty()) { + // The payoff of a time-travel trace: [Out] parameters can be + // rendered filled in, which a live debugger can't easily do. + resolvePendingOuts(top.pending, thread, decode_opt, call.params); + call.args = toCapaArgs(call.params); + } it->second.pop_back(); } } @@ -154,7 +236,11 @@ int wmain(int argc, wchar_t** argv) { }; sweep->SetCallReturnCallback(on_call_return); - sweep->SetReplayFlags(TTD::Replay::ReplayFlags::ReplaySegmentsSequentially); + // Without ReplayAllSegmentsWithoutFiltering the engine only replays segments it + // thinks can hit an event, and a call/return callback alone doesn't qualify -- + // the sweep then completes instantly having seen nothing. + sweep->SetReplayFlags(TTD::Replay::ReplayFlags::ReplaySegmentsSequentially | + TTD::Replay::ReplayFlags::ReplayAllSegmentsWithoutFiltering); sweep->SetPosition(TTD::Replay::Position::Min); std::cerr << "[+] Beginning execution sweep...\n"; sweep->ReplayForward(); @@ -162,7 +248,8 @@ int wmain(int argc, wchar_t** argv) { if (limit_hit) { std::cerr << "[!] Reached --max-calls limit (" << opt.max_calls << ")\n"; } - std::cerr << "[+] Recorded " << g_report.process.calls.size() << " API calls\n"; + std::cerr << "[+] Recorded " << g_report.process.calls.size() << " API calls from " + << events_seen << " call/return events\n"; writeReport(opt.output); diff --git a/ttd/src/ttdutils.cpp b/ttd/src/ttdutils.cpp index eb8a28c..fb5b7c1 100644 --- a/ttd/src/ttdutils.cpp +++ b/ttd/src/ttdutils.cpp @@ -48,13 +48,30 @@ namespace ttdcapa { return resolvedTraceModuleExports; } + std::string convertWstringToString(const std::wstring& ws) { + if (ws.empty()) { + return {}; + } + int needed = ::WideCharToMultiByte(CP_UTF8, 0, ws.data(), static_cast(ws.size()), nullptr, 0, nullptr, nullptr); + if (needed <= 0) { + return {}; + } + std::string out(static_cast(needed), '\0'); + ::WideCharToMultiByte(CP_UTF8, 0, ws.data(), static_cast(ws.size()), out.data(), needed, nullptr, nullptr); + return out; + } + size_t readMemory(TTD::Replay::UniqueCursor* cursor, TTD::GuestAddress addr, void* dest, unsigned __int64 size) { TTD::Replay::MemoryBuffer memoryBuffer = cursor->get()->QueryMemoryBuffer(addr, TTD::BufferView{ dest, size }); return memoryBuffer.Memory.Size; } bool initializeTTDEngine(TTD::Replay::UniqueReplayEngine& engine, std::wstring traceFilePath) { - ConsoleErrorReporting reporter; + // The engine holds this pointer for its whole lifetime and calls into it + // whenever it reports an error. A stack local would dangle the moment this + // function returns, and the first derailment during ReplayForward would then + // dispatch a virtual call through reclaimed stack memory. + static ConsoleErrorReporting reporter; engine->RegisterDebugModeAndLogging(TTD::Replay::DebugModeType::None, &reporter); if (!engine->Initialize(traceFilePath.c_str())) { @@ -132,6 +149,105 @@ namespace ttdcapa { return std::nullopt; } + namespace { + // Shared by both typed readers: an address below the first 64 KiB is never a + // valid user-mode string pointer, and treating one as such is how a flags + // DWORD ends up rendered as text. + constexpr uint64_t kMinStringAddr = 0x10000; + + bool isPlausibleTextByte(unsigned char c) { + return c == '\t' || c == '\r' || c == '\n' || (c >= 0x20 && c != 0x7f); + } + + // The -A entry points take strings in the ANSI code page, so their high + // bytes are not UTF-8. Emitting them raw would produce a report that the + // JSON serializer refuses to write, so transcode before anything else sees + // them. Pure-ASCII input (the overwhelming majority) short-circuits. + std::string ansiToUtf8(std::string bytes) { + bool ascii = true; + for (unsigned char c : bytes) { + if (c >= 0x80) { + ascii = false; + break; + } + } + if (ascii) { + return bytes; + } + + int wide = ::MultiByteToWideChar(CP_ACP, 0, bytes.data(), static_cast(bytes.size()), nullptr, 0); + if (wide > 0) { + std::wstring ws(static_cast(wide), L'\0'); + if (::MultiByteToWideChar(CP_ACP, 0, bytes.data(), static_cast(bytes.size()), ws.data(), wide) == wide) { + return convertWstringToString(ws); + } + } + + // Unconvertible: keep the ASCII skeleton rather than dropping the string. + for (char& c : bytes) { + if (static_cast(c) >= 0x80) { + c = '?'; + } + } + return bytes; + } + } // namespace + + std::optional readAnsiString(TTD::Replay::IThreadView const* thread, uint64_t addr, size_t maxChars) { + if (addr < kMinStringAddr) { + return std::nullopt; + } + std::vector buf(maxChars + 1); + auto result = thread->QueryMemoryBuffer(TTD::GuestAddress{ addr }, TTD::BufferView{ buf.data(), maxChars }); + size_t avail = result.Memory.Size; + if (avail == 0) { + return std::nullopt; + } + + std::string s; + for (size_t i = 0; i < avail; ++i) { + unsigned char c = static_cast(buf[i]); + if (c == 0) { + return ansiToUtf8(std::move(s)); + } + if (!isPlausibleTextByte(c)) { + return std::nullopt; // control bytes mean this wasn't a string after all + } + s.push_back(static_cast(c)); + } + // Ran out of readable memory before the terminator; keep what we have as + // long as it looked like text the whole way. + return s.empty() ? std::nullopt : std::optional(ansiToUtf8(std::move(s))); + } + + std::optional readWideString(TTD::Replay::IThreadView const* thread, uint64_t addr, size_t maxChars) { + if (addr < kMinStringAddr) { + return std::nullopt; + } + std::vector buf(maxChars + 1); + auto result = thread->QueryMemoryBuffer( + TTD::GuestAddress{ addr }, TTD::BufferView{ buf.data(), maxChars * sizeof(wchar_t) }); + size_t availChars = result.Memory.Size / sizeof(wchar_t); + if (availChars == 0) { + return std::nullopt; + } + + size_t len = 0; + while (len < availChars && buf[len] != L'\0') { + wchar_t wc = buf[len]; + // Reject C0 controls (other than the usual whitespace) rather than + // emitting mojibake for a pointer that only looked like a string. + if (wc < 0x20 && wc != L'\t' && wc != L'\r' && wc != L'\n') { + return std::nullopt; + } + ++len; + } + if (len == 0) { + return std::string{}; + } + return convertWstringToString(std::wstring(buf.data(), len)); + } + // Capture one candidate argument: a dereferenced string if it points to one, otherwise the raw integer value. ttdcapa::ArgValue captureCallArg(TTD::Replay::IThreadView const* thread, uint64_t value) { if (auto s = tryReadString(thread, value)) { diff --git a/ttd/src/ttdutils.hpp b/ttd/src/ttdutils.hpp index e4ed5af..361c0c4 100644 --- a/ttd/src/ttdutils.hpp +++ b/ttd/src/ttdutils.hpp @@ -14,6 +14,8 @@ #include #include +#include "win32meta.hpp" + class ConsoleErrorReporting : public TTD::ErrorReporting { public: void __fastcall VPrintError(char const* fmt, va_list args) override { @@ -43,13 +45,49 @@ namespace ttdcapa { // One call argument: either an integer or a dereferenced string using ArgValue = std::variant; + // One parameter decoded with the help of the Win32 metadata index. `name` and + // `type` point into the index blob, which outlives every record, so they cost + // nothing to copy. Enum values keep their index rather than their decoded flag + // names so millions of in-memory calls stay cheap; names are resolved once at + // report-write time. + struct DecodedArg { + const char* name = nullptr; + const char* type = nullptr; + win32meta::ArgKind kind = win32meta::ArgKind::Unknown; + uint32_t enum_index = 0xFFFFFFFFu; + uint64_t raw = 0; // the register/stack value as captured + uint64_t deref = 0; // pointee, for PtrToInt and friends + double fval = 0.0; // for Float/Double params (read from XMM) + std::string str; // decoded string contents + std::vector bytes; // bounded buffer preview + bool has_str = false; + bool has_deref = false; + bool has_fval = false; + bool is_out = false; + bool from_return = false; // contents were read at the return position + }; + + // A dereference deferred until the call returns, so [Out] parameters can be + // rendered filled in -- something only a time-travel trace makes easy. + struct PendingOut { + uint16_t param_index = 0; + win32meta::ArgKind kind = win32meta::ArgKind::Unknown; + uint64_t ptr = 0; + uint16_t pointee_size = 0; + win32meta::AuxKind aux_kind = win32meta::AuxKind::None; + int32_t aux_value = 0; + uint64_t in_cap = 0; // caller-supplied upper bound on length, 0 if unknown + }; + struct CallRecord { uint64_t tid = 0; // TTD Thread ID uint64_t seq = 0; // monotonic record order (consistent with timeline order) std::string position; // TTD navigable position "Sequence:Steps" (hex), for WinDbg std::wstring module; // resolved owning module, e.g. "kernel32" (no extension) std::string api; // resolved export name, e.g. "CreateFileA" - std::vector args; + std::vector args; // flat view consumed by capa (ints and strings) + std::vector params; // rich view; only meaningful when `metadata` is set + bool metadata = false; // args came from a real signature, not the heuristic bool has_ret = false; uint64_t ret = 0; }; @@ -78,12 +116,22 @@ namespace ttdcapa { // TTD memory read utility function size_t readMemory(TTD::Replay::UniqueCursor* cursor, TTD::GuestAddress addr, void* dest, unsigned __int64 size); + // UTF-16 -> UTF-8, for the wide strings TTD and the PE headers hand back + std::string convertWstringToString(const std::wstring& ws); + // Initializes the TTD engine based off a given trace file (.run) path bool initializeTTDEngine(TTD::Replay::UniqueReplayEngine& engine, std::wstring trace_file_path); // Attempts to interpret the memory at a certain address as a string. If not a string, will return null std::optional tryReadString(TTD::Replay::IThreadView const* thread, uint64_t addr); + // Read a NUL-terminated string the metadata told us is really there. Unlike + // tryReadString these do not guess: no minimum length, and the wide reader + // converts real UTF-16 (not just its ASCII subset) to UTF-8. Returns nullopt + // only when the memory is unreadable or the bytes aren't a plausible string. + std::optional readAnsiString(TTD::Replay::IThreadView const* thread, uint64_t addr, size_t maxChars = 512); + std::optional readWideString(TTD::Replay::IThreadView const* thread, uint64_t addr, size_t maxChars = 512); + // Attempts to capture an argument as a string. If it doesn't look like a valid string, this function will return the same argument value ArgValue captureCallArg(TTD::Replay::IThreadView const* thread, uint64_t value); } diff --git a/ttd/src/utils.cpp b/ttd/src/utils.cpp index f8a98e3..c62953e 100644 --- a/ttd/src/utils.cpp +++ b/ttd/src/utils.cpp @@ -56,18 +56,6 @@ namespace ttdcapa { return result; } - std::string convertWstringToString(const std::wstring& ws) { - if (ws.empty()) { - return {}; - } - int needed = ::WideCharToMultiByte(CP_UTF8, 0, ws.data(), static_cast(ws.size()), nullptr, 0, nullptr, nullptr); - if (needed <= 0) { - return {}; - } - std::string out(static_cast(needed), '\0'); - ::WideCharToMultiByte(CP_UTF8, 0, ws.data(), static_cast(ws.size()), out.data(), needed, nullptr, nullptr); - return out; - } } // namespace SampleHashes hashFile(const std::filesystem::path& path) { @@ -178,6 +166,18 @@ namespace ttdcapa { else if (a == L"--with-stack-args") { opt.with_stack_args = true; } + else if (a == L"--win32-index" && i + 1 < argc) { + opt.win32_index = argv[++i]; + } + else if (a == L"--no-metadata") { + opt.no_metadata = true; + } + else if (a == L"--max-buffer" && i + 1 < argc) { + opt.max_buffer = static_cast(std::wcstoull(argv[++i], nullptr, 10)); + } + else if (a == L"--dump-sig" && i + 1 < argc) { + opt.dump_sig = convertWstringToString(argv[++i]); + } else if (!a.empty() && a[0] == L'-') { std::cerr << "unknown option\n"; return false; @@ -190,7 +190,8 @@ namespace ttdcapa { return false; } } - return !opt.trace.empty(); + // --dump-sig is a metadata-only debug mode, so it doesn't need a trace. + return !opt.trace.empty() || !opt.dump_sig.empty(); } bool writeReport(std::filesystem::path outputFilePath) { @@ -239,6 +240,9 @@ namespace ttdcapa { call["module"] = moduleStr; call["api"] = callRecord.api; + // Always an array, even when empty -- a zero-parameter function is a + // real result, not a missing field. + call["args"] = json::array(); for (ArgValue& argValue : callRecord.args) { if (std::holds_alternative(argValue)) { call["args"].push_back(std::get(argValue)); @@ -247,6 +251,46 @@ namespace ttdcapa { } } + // The rich, metadata-derived view. capa matches on "args"; this is for + // ttd-timeline and human triage, so it can afford to be verbose. Absent + // entirely when we had no signature, so consumers can tell "no + // parameters" apart from "we didn't know". + if (callRecord.metadata) { + call["params"] = json::array(); + } + for (DecodedArg& decoded : callRecord.params) { + json param; + param["name"] = decoded.name ? decoded.name : ""; + param["type"] = decoded.type ? decoded.type : ""; + param["kind"] = win32meta::kindName(decoded.kind); + param["value"] = decoded.raw; + if (decoded.has_str) { + param["str"] = decoded.str; + } + if (decoded.has_deref) { + param["deref"] = decoded.deref; + } + if (decoded.has_fval) { + param["float"] = decoded.fval; + } + if (decoded.enum_index != 0xFFFFFFFFu) { + std::vector names = win32meta::index().decodeEnum(decoded.enum_index, decoded.raw); + if (!names.empty()) { + param["flags"] = names; + } + } + if (!decoded.bytes.empty()) { + param["bytes"] = to_hex(decoded.bytes); + } + if (decoded.is_out) { + param["out"] = true; + } + if (decoded.from_return) { + param["at_return"] = true; + } + call["params"].push_back(std::move(param)); + } + call["ret"] = callRecord.has_ret ? callRecord.ret : 0; process["calls"].push_back(call); @@ -260,8 +304,12 @@ namespace ttdcapa { return false; } - reportFile << report.dump() << std::endl; + // Recovered guest memory can always surprise us with a byte sequence that + // isn't valid UTF-8. Substituting it beats throwing away an entire trace's + // worth of extraction over one bad string. + reportFile << report.dump(-1, ' ', false, nlohmann::json::error_handler_t::replace) << std::endl; reportFile.close(); std::cerr << "DONE!\n"; + return true; } } diff --git a/ttd/src/utils.hpp b/ttd/src/utils.hpp index 71860e7..908d4ab 100644 --- a/ttd/src/utils.hpp +++ b/ttd/src/utils.hpp @@ -31,9 +31,15 @@ namespace ttdcapa { struct Options { std::filesystem::path trace; - std::filesystem::path sample; // optional on-disk sample for hashing - std::filesystem::path output; // empty will output to stdout - uint64_t max_calls = 0; // 0 means unlimited + std::filesystem::path sample; // optional on-disk sample for hashing + std::filesystem::path output; // empty will output to stdout + std::filesystem::path win32_index; // empty means search the default locations + std::string dump_sig; // print one signature and exit; no trace needed + uint64_t max_calls = 0; // 0 means unlimited + size_t max_buffer = 256; // bytes kept from any one counted buffer + bool no_metadata = false; // force the pre-metadata heuristic capture + // Only affects calls with no metadata: blindly grab four extra stack slots. + // Functions we have a signature for always capture their true arity. bool with_stack_args = false; }; diff --git a/ttd/src/win32meta.cpp b/ttd/src/win32meta.cpp new file mode 100644 index 0000000..e5df5fc --- /dev/null +++ b/ttd/src/win32meta.cpp @@ -0,0 +1,277 @@ +#include "win32meta.hpp" + +#ifndef WIN32_LEAN_AND_MEAN +#define WIN32_LEAN_AND_MEAN +#endif +#include + +#include +#include +#include + +namespace ttdcapa::win32meta { + namespace { + constexpr char kMagic[8] = { 'W', '3', '2', 'I', 'D', 'X', '0', '1' }; + constexpr uint32_t kFormatVersion = 1; + + // Record sizes must match the struct.pack formats in tools/build-win32-index.py. + constexpr size_t kHeaderSize = 32; + constexpr size_t kFuncRecSize = 16; + constexpr size_t kParamRecSize = 24; + constexpr size_t kEnumRecSize = 16; + constexpr size_t kEnumValRecSize = 16; + + template + T readAt(const uint8_t* p, size_t off) { + T v{}; + std::memcpy(&v, p + off, sizeof(T)); + return v; + } + + int popcount64(uint64_t v) { + int n = 0; + while (v) { + v &= v - 1; + ++n; + } + return n; + } + } // namespace + + bool Index::load(const std::filesystem::path& path, std::string& error) { + std::ifstream f(path, std::ios::binary); + if (!f) { + error = "cannot open index"; + return false; + } + std::vector blob((std::istreambuf_iterator(f)), std::istreambuf_iterator()); + if (blob.size() < kHeaderSize || std::memcmp(blob.data(), kMagic, sizeof(kMagic)) != 0) { + error = "not a win32 index file (bad magic)"; + return false; + } + + const uint8_t* p = blob.data(); + uint32_t version = readAt(p, 8); + if (version != kFormatVersion) { + error = "index format version " + std::to_string(version) + + ", expected " + std::to_string(kFormatVersion) + " (regenerate with tools/build-win32-index.py)"; + return false; + } + + uint32_t funcCount = readAt(p, 12); + uint32_t paramCount = readAt(p, 16); + uint32_t enumCount = readAt(p, 20); + uint32_t enumValCount = readAt(p, 24); + uint32_t strtabSize = readAt(p, 28); + + size_t funcOff = kHeaderSize; + size_t paramOff = funcOff + static_cast(funcCount) * kFuncRecSize; + size_t enumOff = paramOff + static_cast(paramCount) * kParamRecSize; + size_t enumValOff = enumOff + static_cast(enumCount) * kEnumRecSize; + size_t strOff = enumValOff + static_cast(enumValCount) * kEnumValRecSize; + if (strOff + strtabSize != blob.size()) { + error = "index is truncated or corrupt"; + return false; + } + // Every string offset is dereferenced without a bounds check below, so the + // pool must be NUL-terminated for that to be safe. + if (strtabSize == 0 || blob[blob.size() - 1] != 0) { + error = "index string pool is not NUL-terminated"; + return false; + } + + blob_ = std::move(blob); + p = blob_.data(); + const char* strtab = reinterpret_cast(p + strOff); + auto str = [&](uint32_t off) -> const char* { + return off < strtabSize ? strtab + off : ""; + }; + + params_.resize(paramCount); + for (uint32_t i = 0; i < paramCount; ++i) { + size_t o = paramOff + static_cast(i) * kParamRecSize; + ParamSig& ps = params_[i]; + ps.name = str(readAt(p, o)); + ps.type = str(readAt(p, o + 4)); + ps.kind = static_cast(p[o + 8]); + ps.attrs = p[o + 9]; + ps.slot = p[o + 10]; + ps.auxKind = static_cast(p[o + 11]); + ps.auxValue = readAt(p, o + 12); + ps.enumIndex = readAt(p, o + 16); + ps.pointeeSize = readAt(p, o + 20); + if (ps.enumIndex != 0xFFFFFFFFu && ps.enumIndex >= enumCount) { + ps.enumIndex = 0xFFFFFFFFu; + } + } + + funcs_.resize(funcCount); + byName_.reserve(funcCount * 2); + for (uint32_t i = 0; i < funcCount; ++i) { + size_t o = funcOff + static_cast(i) * kFuncRecSize; + FuncSig& fs = funcs_[i]; + fs.name = str(readAt(p, o)); + fs.dll = str(readAt(p, o + 4)); + uint32_t firstParam = readAt(p, o + 8); + fs.paramCount = p[o + 12]; + fs.flags = p[o + 13]; + if (static_cast(firstParam) + fs.paramCount > params_.size()) { + error = "index parameter range out of bounds"; + return false; + } + fs.params = fs.paramCount ? ¶ms_[firstParam] : nullptr; + byName_.emplace(std::string_view(fs.name), i); + } + + enumValues_.resize(enumValCount); + for (uint32_t i = 0; i < enumValCount; ++i) { + size_t o = enumValOff + static_cast(i) * kEnumValRecSize; + enumValues_[i].name = str(readAt(p, o)); + enumValues_[i].value = readAt(p, o + 8); + } + + enums_.resize(enumCount); + for (uint32_t i = 0; i < enumCount; ++i) { + size_t o = enumOff + static_cast(i) * kEnumRecSize; + EnumTable& et = enums_[i]; + et.name = str(readAt(p, o)); + et.valueOffset = readAt(p, o + 4); + et.valueCount = readAt(p, o + 8); + et.isFlags = p[o + 12] != 0; + et.width = p[o + 13]; + if (static_cast(et.valueOffset) + et.valueCount > enumValues_.size()) { + et.valueOffset = 0; + et.valueCount = 0; + } + } + + path_ = path; + return true; + } + + const FuncSig* Index::lookup(std::string_view api) const { + auto it = byName_.find(api); + return it == byName_.end() ? nullptr : &funcs_[it->second]; + } + + const char* Index::enumName(uint32_t enumIndex) const { + return enumIndex < enums_.size() ? enums_[enumIndex].name : ""; + } + + std::vector Index::decodeEnum(uint32_t enumIndex, uint64_t value) const { + std::vector out; + if (enumIndex >= enums_.size()) { + return out; + } + const EnumTable& et = enums_[enumIndex]; + // The captured value is a full 64-bit register; mask to the enum's real width + // so sign-extension and upper garbage don't defeat the comparisons. + uint64_t mask = et.width >= 8 ? ~0ull : ((1ull << (et.width * 8)) - 1); + uint64_t v = value & mask; + + const EnumValue* vals = enumValues_.data() + et.valueOffset; + for (uint32_t i = 0; i < et.valueCount; ++i) { + if ((static_cast(vals[i].value) & mask) == v) { + out.emplace_back(vals[i].name); + return out; // exact match wins, flags or not + } + } + if (!et.isFlags || v == 0) { + return out; + } + + // Greedy decomposition: consume the widest matching bit groups first so + // composites like GENERIC_WRITE beat their individual constituent bits. + std::vector order(et.valueCount); + for (uint32_t i = 0; i < et.valueCount; ++i) { + order[i] = i; + } + std::sort(order.begin(), order.end(), [&](uint32_t a, uint32_t b) { + return popcount64(static_cast(vals[a].value) & mask) > + popcount64(static_cast(vals[b].value) & mask); + }); + + uint64_t remaining = v; + for (uint32_t i : order) { + uint64_t bits = static_cast(vals[i].value) & mask; + if (bits != 0 && (remaining & bits) == bits) { + out.emplace_back(vals[i].name); + remaining &= ~bits; + } + } + if (remaining != 0) { + char buf[32]; + std::snprintf(buf, sizeof(buf), "0x%llx", static_cast(remaining)); + out.emplace_back(buf); + } + return out; + } + + Index& index() { + static Index instance; + return instance; + } + + namespace { + std::filesystem::path executableDir() { + wchar_t buf[MAX_PATH * 4]; + DWORD n = ::GetModuleFileNameW(nullptr, buf, static_cast(std::size(buf))); + if (n == 0 || n >= std::size(buf)) { + return {}; + } + return std::filesystem::path(buf, buf + n).parent_path(); + } + } // namespace + + bool loadIndex(const std::filesystem::path& explicitPath, std::string& error) { + std::vector candidates; + if (!explicitPath.empty()) { + candidates.push_back(explicitPath); + } else { + std::filesystem::path dir = executableDir(); + if (!dir.empty()) { + candidates.push_back(dir / L"win32-index.bin"); + // running straight out of ttd\bin\\\ during development + candidates.push_back(dir / L".." / L".." / L".." / L"data" / L"win32-index.bin"); + } + } + + std::error_code ec; + for (const auto& c : candidates) { + if (!std::filesystem::exists(c, ec)) { + continue; + } + if (index().load(c, error)) { + return true; + } + return false; // found but unusable: surface the real reason + } + error = "win32-index.bin not found (run tools/build-win32-index.py, or pass --win32-index)"; + return false; + } + + const char* kindName(ArgKind kind) { + switch (kind) { + case ArgKind::Integer: return "int"; + case ArgKind::Bool: return "bool"; + case ArgKind::Handle: return "handle"; + case ArgKind::Enum: return "enum"; + case ArgKind::Float: return "float"; + case ArgKind::Double: return "double"; + case ArgKind::AnsiString: return "str"; + case ArgKind::WideString: return "wstr"; + case ArgKind::AnsiBuffer: return "strbuf"; + case ArgKind::WideBuffer: return "wstrbuf"; + case ArgKind::ByteBuffer: return "buf"; + case ArgKind::PtrToInt: return "int*"; + case ArgKind::StructPtr: return "struct*"; + case ArgKind::FuncPtr: return "fnptr"; + case ArgKind::Guid: return "guid"; + case ArgKind::Pointer: return "ptr"; + case ArgKind::PtrToAnsiString: return "str*"; + case ArgKind::PtrToWideString: return "wstr*"; + case ArgKind::Unknown: + default: return "unknown"; + } + } +} // namespace ttdcapa::win32meta diff --git a/ttd/src/win32meta.hpp b/ttd/src/win32meta.hpp new file mode 100644 index 0000000..e200455 --- /dev/null +++ b/ttd/src/win32meta.hpp @@ -0,0 +1,156 @@ +#ifndef WIN32META_HPP +#define WIN32META_HPP + +// Loader for the pre-baked Win32 API metadata index (win32-index.bin) produced by +// tools/build-win32-index.py from the win32json submodule. +// +// The index answers one question per recorded call: "given this export name, what +// are its parameters?" -- count, names, types, direction, and how to decode each +// one. That turns the extractor's blind 4-register grab into an exact capture. +// +// Lookup is keyed on the bare function name. Module is carried for display only: +// API sets mean the metadata's "CreateFileW -> KERNEL32.dll" shows up in a trace as +// KERNELBASE.dll!CreateFileW, so matching on module would lose most calls. +// See WIN32JSON-TTD-INTEGRATION-NOTES.md section 4. + +#include +#include +#include +#include +#include +#include + +namespace ttdcapa::win32meta { + + // How to decode one parameter. Values must match the K_* constants in + // tools/build-win32-index.py. + enum class ArgKind : uint8_t { + Unknown = 0, + Integer, // plain scalar, never dereferenced + Bool, + Handle, // opaque HANDLE/HKEY/HWND; pointer-sized but not a pointer + Enum, // scalar with a symbolic value table (see enumIndex) + Float, // 4-byte float, arrives in XMM + Double, // 8-byte float, arrives in XMM + AnsiString, // char*, NUL-terminated + WideString, // wchar_t*, NUL-terminated + AnsiBuffer, // char[], length from aux + WideBuffer, // wchar_t[], length from aux + ByteBuffer, // void*/byte[], length from aux + PtrToInt, // pointer to a pointeeSize-byte scalar + StructPtr, // pointer to a struct/union (not expanded in v1) + FuncPtr, + Guid, // pointer to a 16-byte GUID + Pointer, // opaque pointer, contents unknown + PtrToAnsiString, // char** -- out-param that receives an allocated string + PtrToWideString, // wchar_t** + }; + + // Bitmask of a parameter's direction/nullability attributes. + enum ParamAttr : uint8_t { + AttrIn = 0x01, + AttrOut = 0x02, + AttrOptional = 0x04, + AttrConst = 0x08, + AttrReserved = 0x10, + AttrNotNulTerminated = 0x20, + AttrNulNulTerminated = 0x40, + AttrComOutPtr = 0x80, + }; + + // Where a buffer parameter's length comes from. + enum class AuxKind : uint8_t { + None = 0, + BytesFromParam, // auxValue is the index of a parameter holding a byte count + CountFromParam, // auxValue is the index of a parameter holding an element count + CountConst, // auxValue is the element count itself + }; + + enum FuncFlag : uint8_t { + FlagHiddenRetPtr = 0x01, // returns a large aggregate; RCX is the hidden return buffer + FlagUnsupported = 0x02, // at least one parameter could not be classified + FlagSetLastError = 0x04, + }; + + struct ParamSig { + const char* name = ""; + const char* type = ""; + ArgKind kind = ArgKind::Unknown; + uint8_t attrs = 0; + uint8_t slot = 0; // positional ABI slot, already shifted for FlagHiddenRetPtr + AuxKind auxKind = AuxKind::None; + int32_t auxValue = 0; + uint32_t enumIndex = 0xFFFFFFFFu; + uint16_t pointeeSize = 0; // bytes per pointee/element, 0 if unknown + + bool isIn() const { return (attrs & AttrIn) != 0; } + bool isOut() const { return (attrs & AttrOut) != 0; } + bool isFloat() const { return kind == ArgKind::Float || kind == ArgKind::Double; } + bool hasEnum() const { return enumIndex != 0xFFFFFFFFu; } + }; + + struct FuncSig { + const char* name = ""; + const char* dll = ""; + const ParamSig* params = nullptr; + uint8_t paramCount = 0; + uint8_t flags = 0; + + bool unsupported() const { return (flags & FlagUnsupported) != 0; } + bool hiddenRetPtr() const { return (flags & FlagHiddenRetPtr) != 0; } + }; + + // Loaded once at startup and read-only thereafter, so the replay sweep can hit + // it from the call callback without synchronisation. + class Index { + public: + bool load(const std::filesystem::path& path, std::string& error); + bool loaded() const { return !funcs_.empty(); } + size_t functionCount() const { return funcs_.size(); } + const std::filesystem::path& path() const { return path_; } + + // Bare export name, e.g. "CreateFileW". Returns nullptr when unknown, which + // is the caller's cue to fall back to the heuristic capture. + const FuncSig* lookup(std::string_view api) const; + + const char* enumName(uint32_t enumIndex) const; + + // Symbolic names for `value`: an exact match if one exists, otherwise a + // greedy bit decomposition for flag enums. Empty when nothing matches. + std::vector decodeEnum(uint32_t enumIndex, uint64_t value) const; + + private: + struct EnumTable { + const char* name = ""; + uint32_t valueOffset = 0; + uint32_t valueCount = 0; + bool isFlags = false; + uint8_t width = 4; + }; + struct EnumValue { + const char* name = ""; + int64_t value = 0; + }; + + std::vector blob_; + std::vector funcs_; + std::vector params_; + std::vector enums_; + std::vector enumValues_; + std::unordered_map byName_; + std::filesystem::path path_; + }; + + // Process-wide instance; empty until loadIndex() succeeds. + Index& index(); + + // Try `explicitPath` if non-empty, else the conventional locations next to the + // executable and in the source tree. Returns false with `error` set; the caller + // is expected to warn and continue in heuristic mode. + bool loadIndex(const std::filesystem::path& explicitPath, std::string& error); + + const char* kindName(ArgKind kind); + +} // namespace ttdcapa::win32meta + +#endif diff --git a/ttd/ttdcapa-extract.vcxproj b/ttd/ttdcapa-extract.vcxproj index 9df1f3a..b8de830 100644 --- a/ttd/ttdcapa-extract.vcxproj +++ b/ttd/ttdcapa-extract.vcxproj @@ -70,15 +70,19 @@ + + + + @@ -101,6 +105,13 @@ + + + + + diff --git a/ttd/ttdcapa-extract.vcxproj.filters b/ttd/ttdcapa-extract.vcxproj.filters index 8c6b89d..ecf34c1 100644 --- a/ttd/ttdcapa-extract.vcxproj.filters +++ b/ttd/ttdcapa-extract.vcxproj.filters @@ -1,9 +1,15 @@  + + Source Files + Source Files + + Source Files + Source Files @@ -15,9 +21,15 @@ + + Header Files + Header Files + + Header Files + Header Files diff --git a/win32json b/win32json new file mode 160000 index 0000000..071df49 --- /dev/null +++ b/win32json @@ -0,0 +1 @@ +Subproject commit 071df4995290b72320010ecfc2fca51946b5c6b4