mirror of
https://github.com/HullaBrian/ttd-capa
synced 2026-08-09 12:08:34 +00:00
Sync metadata-driven parameter decoding build from author
Win32 metadata index (win32json) drives per-call signature lookup so arguments are captured at their true arity and type, [Out] params are re-read at the call's return position, and buffers are dumped as bytes.
This commit is contained in:
@@ -1,3 +1,6 @@
|
||||
[submodule "capa"]
|
||||
path = capa
|
||||
url = https://github.com/HullaBrian/capa
|
||||
[submodule "win32json"]
|
||||
path = win32json
|
||||
url = https://github.com/marlersoft/win32json
|
||||
|
||||
@@ -30,6 +30,42 @@ the entire trace. Every time a call occurs, ttd-capa checks if the call target i
|
||||
with the associate module, function name, parameters, and return value. Additionally, ttd-capa automatically attempts to resolve function parameters as strings,
|
||||
which has the potential to significantly increase quick wins during malware analysis.
|
||||
|
||||
## Metadata-driven parameter decoding
|
||||
Knowing an API's name is not the same as knowing its signature. Without one, all an extractor can do is grab the four
|
||||
x64 argument registers and guess at each value - so `CloseHandle(hObject)` gets recorded with four arguments, three of
|
||||
which are whatever the caller happened to leave in RDX/R8/R9, and any integer that happens to point at printable bytes
|
||||
gets mistaken for a string.
|
||||
|
||||
ttd-capa fixes that with Microsoft's own Win32 API metadata, vendored as the [win32json](https://github.com/marlersoft/win32json)
|
||||
submodule. `tools/build-win32-index.py` flattens those ~300 JSON files into a compact binary index
|
||||
(`ttd/data/win32-index.bin`, ~2.7 MB) keyed by export name, which the extractor memory-maps at startup. For every call
|
||||
whose export is in the index, ttd-capa now knows:
|
||||
|
||||
- **the real parameter count**, so exactly that many arguments are captured - including the ones past RCX/RDX/R8/R9
|
||||
(`CreateProcessW` has ten) and none of the register residue that used to masquerade as arguments
|
||||
- **each parameter's type**, so `PSTR` is decoded as ANSI and `PWSTR` as UTF-16 rather than by trial, and values typed
|
||||
as `HANDLE`, enums, or plain integers are never dereferenced at all
|
||||
- **direction** - `[Out]` parameters are re-read at the call's *return* position, so `lpNumberOfBytesRead` and friends
|
||||
are rendered filled in. This is something a live debugger can't easily do, and it's where a time-travel trace earns
|
||||
its keep
|
||||
- **buffer lengths**, from `MemorySize(BytesParamIndex)` and array count parameters, so `InternetReadFile`'s `lpBuffer`
|
||||
can be captured at its true length once the callee has written it
|
||||
- **enum and flag tables**, so `dwCreationDisposition=3` renders as `CREATE_ALWAYS`
|
||||
|
||||
The decoded view lands in a new `params` array on each call in the report. The `args` array capa matches against keeps
|
||||
its original shape - it just has the correct arity now, plus any strings recovered from `[Out]` parameters. Symbolic
|
||||
flag names are deliberately *not* pushed into `args`, since existing rules match flags numerically.
|
||||
|
||||
Coverage is the public Windows SDK, which is what the metadata documents: roughly the top 15% of calls in a typical
|
||||
trace by volume, but the great majority of the *interesting* ones. Calls with no signature - `ntdll` internals, CRT
|
||||
helpers like `memset` - fall back to the original four-register heuristic unchanged.
|
||||
|
||||
To regenerate the index (only needed after bumping the `win32json` submodule):
|
||||
|
||||
```powershell
|
||||
python tools\build-win32-index.py
|
||||
```
|
||||
|
||||
# Prerequisites
|
||||
- Windows
|
||||
- v145 for Microsoft C++ Build Tools
|
||||
@@ -37,11 +73,18 @@ which has the potential to significantly increase quick wins during malware anal
|
||||
- Python 3.10+ (for CAPA)
|
||||
|
||||
# Building ttd-capa
|
||||
Clone with submodules (`git clone --recursive`, or `git submodule update --init` in an existing clone) - the
|
||||
`win32json` submodule supplies the API metadata.
|
||||
|
||||
1. Open `ttd/ttdcapa-extract.sln` in Visual Studio
|
||||
2. Ensure that the required nuget packages (`Microsoft.TimeTravelDebugging.Apis` and `nlohmann.json`) are installed
|
||||
3. Set the build mode to `x64` and `Release`
|
||||
4. Navigate to `Build > Build Solution` in order to begin the build
|
||||
|
||||
The build copies `ttd/data/win32-index.bin` next to the executable. If it's missing, run
|
||||
`python tools\build-win32-index.py` to generate it; without it the extractor still runs, but falls back to heuristic
|
||||
argument capture for every call.
|
||||
|
||||
After the build, ttd-capa cannot run properly without Microsoft's `TTDReplay.dll` and `TTDReplayCPU.dll` being in the same directory as `ttdcapa-extract.exe`.
|
||||
To get those DLLs, ensure you have WinDbg instealled already. Then, run the following PowerShell command to find the DLL location on your system:
|
||||
|
||||
@@ -85,7 +128,12 @@ python ttd-capa.py <trace.run> <rules-dir> --sample sample.exe -- -vv
|
||||
Useful flags:
|
||||
- `--extractor <path>` - specify a path to the ttd-capa extractor executable
|
||||
- `--max-calls N` - cap huge traces to certain number of calls
|
||||
- `--with-stack-args` - captures parameters for function calls that are on the stack (capture more than 5+ parameters for function calls)
|
||||
- `--with-stack-args` - for calls with *no* Win32 metadata, also grab four stack slots past the register arguments.
|
||||
Calls we have a signature for always capture their true arity, stack parameters included, so this only affects the
|
||||
heuristic fallback path
|
||||
- `--win32-index <path>` - use a specific `win32-index.bin` instead of the one next to the extractor
|
||||
- `--no-metadata` - disable metadata-driven decoding and use the original four-register heuristic everywhere
|
||||
- `--max-buffer N` - bytes to keep from any one captured buffer (default 256)
|
||||
- `--keep-json` - keep the generated JSON report after the Python script runs
|
||||
|
||||
## Manual
|
||||
@@ -129,8 +177,14 @@ python ttd-timeline.py <JSON REPORT PATH> --calls
|
||||
|
||||
# Limitations
|
||||
- Only x64 traces are supported at the moment (no x86 or ARM)
|
||||
- Argument captures are heuristic, so errors may occur
|
||||
- Only the functions directly exported by loaded modules are logged in the JSON report
|
||||
- Argument decoding is exact only for APIs covered by the Win32 metadata (public SDK surface). `ntdll` internals,
|
||||
undocumented APIs, and CRT helpers fall back to the four-register heuristic, so their arguments may still be wrong
|
||||
- COM interface methods are not resolved; the metadata has no vtable indices, and the call target is a vtable slot
|
||||
rather than a named export
|
||||
- Variadic functions (`printf`-style) have no statically knowable argument count, so only their fixed parameters are
|
||||
decoded
|
||||
- Structure parameters are recorded as pointers; fields are not expanded
|
||||
|
||||
# Verifying the backend in isolation
|
||||
`tests/test_ttd_extractor.py` loads a report and dumps every feature per scope - helpful to confirm expected `API`/`Number`/`String` features:
|
||||
|
||||
@@ -0,0 +1,641 @@
|
||||
"""
|
||||
Flatten the win32json Win32 API metadata into a compact binary index that
|
||||
ttdcapa-extract loads at startup to decode call parameters.
|
||||
|
||||
The metadata (https://github.com/marlersoft/win32json, vendored as the `win32json`
|
||||
submodule) is ~66 MB of JSON across ~300 files. Parsing that at debug time would be
|
||||
absurd, so we pre-bake it once: resolve the type graph, classify every parameter
|
||||
into an x64 ABI slot plus a small "decode kind" the C++ side can switch on, and
|
||||
write a single mmap-friendly blob.
|
||||
|
||||
python tools/build-win32-index.py [win32json/api] [-o ttd/data/win32-index.bin]
|
||||
|
||||
Regenerate whenever the win32json submodule is bumped. See
|
||||
WIN32JSON-TTD-INTEGRATION-NOTES.md for what the metadata does and does not give us
|
||||
(short version: types and semantics yes, ABI classification is ours -- that's the
|
||||
`SLOT_*` / `classify_param` logic below).
|
||||
"""
|
||||
import io
|
||||
import os
|
||||
import sys
|
||||
import json
|
||||
import glob
|
||||
import struct
|
||||
import argparse
|
||||
from pathlib import Path
|
||||
|
||||
MAGIC = b"W32IDX01"
|
||||
FORMAT_VERSION = 1
|
||||
|
||||
# --- decode kinds; keep in sync with ArgKind in ttd/src/win32meta.hpp ----------
|
||||
K_UNKNOWN = 0
|
||||
K_INTEGER = 1
|
||||
K_BOOL = 2
|
||||
K_HANDLE = 3
|
||||
K_ENUM = 4
|
||||
K_FLOAT = 5
|
||||
K_DOUBLE = 6
|
||||
K_ANSI_STRING = 7
|
||||
K_WIDE_STRING = 8
|
||||
K_ANSI_BUFFER = 9
|
||||
K_WIDE_BUFFER = 10
|
||||
K_BYTE_BUFFER = 11
|
||||
K_PTR_TO_INT = 12
|
||||
K_STRUCT_PTR = 13
|
||||
K_FUNC_PTR = 14
|
||||
K_GUID = 15
|
||||
K_POINTER = 16
|
||||
K_PTR_TO_ANSI_STRING = 17
|
||||
K_PTR_TO_WIDE_STRING = 18
|
||||
|
||||
# --- parameter attribute bits; keep in sync with ParamAttr in win32meta.hpp ---
|
||||
A_IN = 0x01
|
||||
A_OUT = 0x02
|
||||
A_OPTIONAL = 0x04
|
||||
A_CONST = 0x08
|
||||
A_RESERVED = 0x10
|
||||
A_NOT_NUL_TERM = 0x20
|
||||
A_NULNUL_TERM = 0x40
|
||||
A_COM_OUT_PTR = 0x80
|
||||
|
||||
# --- how a buffer's length is determined; keep in sync with AuxKind -----------
|
||||
AUX_NONE = 0
|
||||
AUX_BYTES_FROM_PARAM = 1
|
||||
AUX_COUNT_FROM_PARAM = 2
|
||||
AUX_COUNT_CONST = 3
|
||||
|
||||
# --- function flags; keep in sync with FuncFlag ------------------------------
|
||||
F_HIDDEN_RET_PTR = 0x01
|
||||
F_UNSUPPORTED = 0x02
|
||||
F_SET_LAST_ERROR = 0x04
|
||||
|
||||
NO_ENUM = 0xFFFFFFFF
|
||||
|
||||
# x64 sizes for the metadata's primitive names
|
||||
NATIVE_SIZE = {
|
||||
"Byte": 1, "SByte": 1, "Boolean": 1,
|
||||
"Int16": 2, "UInt16": 2, "Char": 2,
|
||||
"Int32": 4, "UInt32": 4, "Single": 4,
|
||||
"Int64": 8, "UInt64": 8, "Double": 8, "IntPtr": 8, "UIntPtr": 8,
|
||||
"Guid": 16,
|
||||
"Void": 0,
|
||||
}
|
||||
NATIVE_INT = {
|
||||
"Byte", "SByte", "Boolean", "Int16", "UInt16", "Char",
|
||||
"Int32", "UInt32", "Int64", "UInt64", "IntPtr", "UIntPtr",
|
||||
}
|
||||
|
||||
# NativeTypedefs that are really strings, not opaque handles. BSTR carries a
|
||||
# FreeFunc (SysFreeString) so the handle heuristic below would otherwise claim it.
|
||||
STRING_TYPEDEFS = {"PSTR": K_ANSI_STRING, "PWSTR": K_WIDE_STRING, "BSTR": K_WIDE_STRING}
|
||||
|
||||
MAX_PARAMS = 255 # param_count is a u8; nothing real comes close (max observed: 18)
|
||||
|
||||
|
||||
class StringTable:
|
||||
"""Deduplicating NUL-terminated string pool. Offset 0 is always ""."""
|
||||
|
||||
def __init__(self):
|
||||
self.buf = bytearray(b"\x00")
|
||||
self.offsets = {"": 0}
|
||||
|
||||
def add(self, s):
|
||||
if s is None:
|
||||
s = ""
|
||||
off = self.offsets.get(s)
|
||||
if off is None:
|
||||
off = len(self.buf)
|
||||
self.buf += s.encode("utf-8") + b"\x00"
|
||||
self.offsets[s] = off
|
||||
return off
|
||||
|
||||
|
||||
class Metadata:
|
||||
"""The win32json type graph, plus the resolution helpers built on top of it."""
|
||||
|
||||
def __init__(self, api_dir):
|
||||
self.types = {} # (api, name) -> type dict
|
||||
self.functions = [] # (api, function dict)
|
||||
self.unicode_aliases = set()
|
||||
files = sorted(glob.glob(os.path.join(api_dir, "*.json")))
|
||||
if not files:
|
||||
sys.exit(f"no .json files under {api_dir}; is the win32json submodule checked out?")
|
||||
for path in files:
|
||||
api = os.path.basename(path)[:-5]
|
||||
with open(path, encoding="utf-8") as f:
|
||||
doc = json.load(f)
|
||||
for t in doc.get("Types", []):
|
||||
# first definition wins; arch-specific duplicates are handled by
|
||||
# SupportedArchitecture, which we resolve to x64 at index time
|
||||
self.types.setdefault((api, t["Name"]), t)
|
||||
for fn in doc.get("Functions", []):
|
||||
self.functions.append((api, fn))
|
||||
self.unicode_aliases.update(doc.get("UnicodeAliases", []))
|
||||
self._size_cache = {}
|
||||
|
||||
def lookup(self, ref):
|
||||
return self.types.get((ref.get("Api"), ref.get("Name")))
|
||||
|
||||
def resolve(self, t, depth=0):
|
||||
"""Follow ApiRef -> NativeTypedef chains to a terminal type node.
|
||||
|
||||
Returns (node, typedef) where `typedef` is the last NativeTypedef we passed
|
||||
through (or None). Callers need the typedef to spot HANDLE/PSTR, which are
|
||||
only distinguishable by name -- their definitions are plain IntPtr/Byte*.
|
||||
"""
|
||||
seen = set()
|
||||
typedef = None
|
||||
while depth < 16:
|
||||
depth += 1
|
||||
if t.get("Kind") != "ApiRef":
|
||||
return t, typedef
|
||||
key = (t.get("Api"), t.get("Name"))
|
||||
if key in seen:
|
||||
return t, typedef
|
||||
seen.add(key)
|
||||
target = self.lookup(t)
|
||||
if target is None:
|
||||
return t, typedef
|
||||
if target.get("Kind") == "NativeTypedef":
|
||||
typedef = target
|
||||
t = target["Def"]
|
||||
continue
|
||||
return target, typedef
|
||||
return t, typedef
|
||||
|
||||
# --- sizeof, needed only to classify by-value aggregates ------------------
|
||||
|
||||
def sizeof(self, t, depth=0):
|
||||
"""x64 size in bytes, or None if it can't be determined.
|
||||
|
||||
Only aggregates passed/returned by value need this (236 params and 18
|
||||
return types across the whole surface), so a None here costs us one
|
||||
function, not correctness everywhere.
|
||||
"""
|
||||
if depth > 24:
|
||||
return None
|
||||
kind = t.get("Kind")
|
||||
if kind == "Native":
|
||||
return NATIVE_SIZE.get(t.get("Name"))
|
||||
if kind in ("PointerTo", "LPArray", "FunctionPointer"):
|
||||
return 8
|
||||
if kind == "Array":
|
||||
child = self.sizeof(t["Child"], depth + 1)
|
||||
count = (t.get("Shape") or {}).get("Size")
|
||||
if child is None or not count:
|
||||
return None
|
||||
return child * count
|
||||
if kind == "ApiRef":
|
||||
node, _ = self.resolve(t)
|
||||
if node.get("Kind") == "ApiRef":
|
||||
return None # unresolvable reference
|
||||
return self.sizeof(node, depth + 1)
|
||||
if kind == "Com":
|
||||
return 8
|
||||
if kind == "Enum":
|
||||
return self.enum_width(t)
|
||||
if kind in ("Struct", "Union"):
|
||||
return self._sizeof_record(t, depth)
|
||||
return None
|
||||
|
||||
def _sizeof_record(self, t, depth):
|
||||
key = id(t)
|
||||
cached = self._size_cache.get(key)
|
||||
if cached is not None:
|
||||
return cached[0]
|
||||
self._size_cache[key] = (None,) # cycle guard
|
||||
declared = t.get("Size") or 0
|
||||
if declared:
|
||||
self._size_cache[key] = (declared,)
|
||||
return declared
|
||||
pack = t.get("PackingSize") or 0
|
||||
offset = 0
|
||||
max_align = 1
|
||||
for field in t.get("Fields", []):
|
||||
fsize = self.sizeof(field["Type"], depth + 1)
|
||||
if fsize is None:
|
||||
self._size_cache[key] = (None,)
|
||||
return None
|
||||
align = min(fsize if fsize in (1, 2, 4, 8, 16) else 8, pack) if pack else fsize
|
||||
align = max(1, min(align if align in (1, 2, 4, 8, 16) else 8, 8))
|
||||
max_align = max(max_align, align)
|
||||
if t["Kind"] == "Union":
|
||||
offset = max(offset, fsize)
|
||||
else:
|
||||
offset = (offset + align - 1) // align * align + fsize
|
||||
size = (offset + max_align - 1) // max_align * max_align if offset else 0
|
||||
self._size_cache[key] = (size,)
|
||||
return size
|
||||
|
||||
def enum_width(self, enum_type):
|
||||
node = enum_type
|
||||
if node.get("Kind") == "ApiRef":
|
||||
node, _ = self.resolve(node)
|
||||
integer_base = node.get("IntegerBase")
|
||||
if integer_base:
|
||||
return NATIVE_SIZE.get(integer_base, 4)
|
||||
return 4
|
||||
|
||||
|
||||
def attr_bits(attrs):
|
||||
"""Fold a param's Attrs list into a bitmask plus its MemorySize source, if any."""
|
||||
bits = 0
|
||||
bytes_param = None
|
||||
for a in attrs:
|
||||
if isinstance(a, dict):
|
||||
if a.get("Kind") == "MemorySize":
|
||||
bytes_param = a.get("BytesParamIndex")
|
||||
continue
|
||||
bits |= {
|
||||
"In": A_IN, "Out": A_OUT, "Optional": A_OPTIONAL, "Const": A_CONST,
|
||||
"Reserved": A_RESERVED, "NotNullTerminated": A_NOT_NUL_TERM,
|
||||
"NullNullTerminated": A_NULNUL_TERM, "ComOutPtr": A_COM_OUT_PTR,
|
||||
}.get(a, 0)
|
||||
return bits, bytes_param
|
||||
|
||||
|
||||
class Param:
|
||||
__slots__ = ("name", "type_name", "kind", "attrs", "slot",
|
||||
"aux_kind", "aux_value", "enum_idx", "pointee_size")
|
||||
|
||||
def __init__(self):
|
||||
self.name = ""
|
||||
self.type_name = ""
|
||||
self.kind = K_UNKNOWN
|
||||
self.attrs = 0
|
||||
self.slot = 0
|
||||
self.aux_kind = AUX_NONE
|
||||
self.aux_value = 0
|
||||
self.enum_idx = NO_ENUM
|
||||
self.pointee_size = 0
|
||||
|
||||
|
||||
def type_display_name(t):
|
||||
"""A short human-readable name for the report/timeline, e.g. "PWSTR", "void*"."""
|
||||
kind = t.get("Kind")
|
||||
if kind == "ApiRef":
|
||||
return t.get("Name", "")
|
||||
if kind == "Native":
|
||||
return t.get("Name", "")
|
||||
if kind == "PointerTo":
|
||||
return type_display_name(t["Child"]) + "*"
|
||||
if kind == "LPArray":
|
||||
return type_display_name(t["Child"]) + "[]"
|
||||
if kind == "FunctionPointer":
|
||||
return "fnptr"
|
||||
return kind or ""
|
||||
|
||||
|
||||
def classify_pointee(md, child):
|
||||
"""Classify what a pointer points at -> (kind, pointee_size).
|
||||
|
||||
Split out because PointerTo and LPArray share it.
|
||||
"""
|
||||
node, typedef = md.resolve(child)
|
||||
if typedef is not None and typedef["Name"] in STRING_TYPEDEFS:
|
||||
# e.g. PWSTR* -- an out-parameter that receives an allocated string
|
||||
return ({K_ANSI_STRING: K_PTR_TO_ANSI_STRING,
|
||||
K_WIDE_STRING: K_PTR_TO_WIDE_STRING}[STRING_TYPEDEFS[typedef["Name"]]], 8)
|
||||
|
||||
kind = node.get("Kind")
|
||||
if kind == "Native":
|
||||
name = node.get("Name")
|
||||
if name in ("Byte", "SByte"):
|
||||
return K_ANSI_BUFFER, 1
|
||||
if name == "Char":
|
||||
return K_WIDE_BUFFER, 2
|
||||
if name == "Void":
|
||||
return K_POINTER, 0
|
||||
if name == "Guid":
|
||||
return K_GUID, 16
|
||||
if name in NATIVE_INT:
|
||||
return K_PTR_TO_INT, NATIVE_SIZE[name]
|
||||
if name in ("Single", "Double"):
|
||||
return K_PTR_TO_INT, NATIVE_SIZE[name]
|
||||
return K_POINTER, 0
|
||||
if kind == "Enum":
|
||||
return K_PTR_TO_INT, md.enum_width(node)
|
||||
if kind in ("Struct", "Union", "Com"):
|
||||
return K_STRUCT_PTR, md.sizeof(node) or 0
|
||||
if kind == "FunctionPointer":
|
||||
return K_FUNC_PTR, 8
|
||||
if kind in ("PointerTo", "LPArray"):
|
||||
return K_PTR_TO_INT, 8 # void** / T** -- at least surface the inner pointer
|
||||
return K_POINTER, 0
|
||||
|
||||
|
||||
def classify_param(md, p, enum_ids):
|
||||
"""Map one metadata parameter onto a decode kind + buffer-length source.
|
||||
|
||||
Returns a Param with everything except `slot` filled in (slot assignment needs
|
||||
whole-signature context and happens in build_function).
|
||||
"""
|
||||
out = Param()
|
||||
out.name = p.get("Name") or ""
|
||||
t = p["Type"]
|
||||
out.type_name = type_display_name(t)
|
||||
out.attrs, bytes_param = attr_bits(p.get("Attrs") or [])
|
||||
|
||||
kind = t.get("Kind")
|
||||
|
||||
if kind == "LPArray":
|
||||
out.kind, out.pointee_size = classify_pointee(md, t["Child"])
|
||||
if out.kind == K_POINTER:
|
||||
out.kind = K_BYTE_BUFFER
|
||||
count_param = t.get("CountParamIndex", -1)
|
||||
count_const = t.get("CountConst", -1)
|
||||
if count_param is not None and count_param >= 0:
|
||||
out.aux_kind, out.aux_value = AUX_COUNT_FROM_PARAM, count_param
|
||||
elif count_const is not None and count_const >= 0:
|
||||
out.aux_kind, out.aux_value = AUX_COUNT_CONST, count_const
|
||||
|
||||
elif kind == "PointerTo":
|
||||
out.kind, out.pointee_size = classify_pointee(md, t["Child"])
|
||||
|
||||
elif kind == "ApiRef":
|
||||
node, typedef = md.resolve(t)
|
||||
if typedef is not None and typedef["Name"] in STRING_TYPEDEFS:
|
||||
out.kind = STRING_TYPEDEFS[typedef["Name"]]
|
||||
out.pointee_size = 1 if out.kind == K_ANSI_STRING else 2
|
||||
elif typedef is not None and is_handle_typedef(typedef):
|
||||
# an opaque kernel/GDI/etc handle: pointer-sized, never dereference it
|
||||
out.kind, out.pointee_size = K_HANDLE, 8
|
||||
else:
|
||||
nkind = node.get("Kind")
|
||||
if nkind == "Enum":
|
||||
out.kind = K_ENUM
|
||||
out.pointee_size = md.enum_width(node)
|
||||
enum_key = (node.get("__api__"), node["Name"])
|
||||
out.enum_idx = enum_ids.get(enum_key, NO_ENUM)
|
||||
elif nkind == "Native":
|
||||
out.kind, out.pointee_size = classify_native(node)
|
||||
elif nkind in ("Struct", "Union"):
|
||||
size = md.sizeof(node)
|
||||
if size in (1, 2, 4, 8):
|
||||
out.kind, out.pointee_size = K_INTEGER, size
|
||||
elif size is None:
|
||||
out.kind = K_UNKNOWN
|
||||
else:
|
||||
# x64: aggregates that aren't 1/2/4/8 bytes go by hidden pointer
|
||||
out.kind, out.pointee_size = K_STRUCT_PTR, size
|
||||
elif nkind == "Com":
|
||||
out.kind, out.pointee_size = K_POINTER, 8
|
||||
elif nkind == "FunctionPointer":
|
||||
out.kind, out.pointee_size = K_FUNC_PTR, 8
|
||||
elif nkind == "PointerTo":
|
||||
out.kind, out.pointee_size = classify_pointee(md, node["Child"])
|
||||
else:
|
||||
out.kind = K_UNKNOWN
|
||||
|
||||
elif kind == "Native":
|
||||
out.kind, out.pointee_size = classify_native(t)
|
||||
if out.kind == K_GUID:
|
||||
out.kind = K_STRUCT_PTR # 16 bytes by value -> hidden pointer on x64
|
||||
|
||||
elif kind == "FunctionPointer":
|
||||
out.kind, out.pointee_size = K_FUNC_PTR, 8
|
||||
|
||||
else:
|
||||
out.kind = K_UNKNOWN
|
||||
|
||||
# a MemorySize attribute always wins: it says this really is a sized buffer
|
||||
if bytes_param is not None and bytes_param >= 0:
|
||||
out.aux_kind, out.aux_value = AUX_BYTES_FROM_PARAM, bytes_param
|
||||
if out.kind in (K_POINTER, K_UNKNOWN, K_STRUCT_PTR):
|
||||
out.kind = K_BYTE_BUFFER
|
||||
out.pointee_size = 1
|
||||
|
||||
return out
|
||||
|
||||
|
||||
def is_handle_typedef(typedef):
|
||||
"""Is this NativeTypedef an opaque handle we must never dereference?
|
||||
|
||||
RAIIFree/InvalidHandleValue mark most of them (HANDLE, HKEY, ...). The rest are
|
||||
pointer-sized H-prefixed typedefs with no lifetime metadata (HWND, HDESK, ...);
|
||||
treating those as integers would be harmless but renders worse.
|
||||
"""
|
||||
if typedef.get("FreeFunc") or typedef.get("InvalidHandleValue") is not None:
|
||||
return True
|
||||
name = typedef["Name"]
|
||||
return (
|
||||
name.startswith("H")
|
||||
and typedef["Def"].get("Kind") == "Native"
|
||||
and typedef["Def"].get("Name") in ("IntPtr", "UIntPtr")
|
||||
)
|
||||
|
||||
|
||||
def classify_native(t):
|
||||
name = t.get("Name")
|
||||
if name == "Single":
|
||||
return K_FLOAT, 4
|
||||
if name == "Double":
|
||||
return K_DOUBLE, 8
|
||||
if name == "Boolean":
|
||||
return K_BOOL, 1
|
||||
if name == "Guid":
|
||||
return K_GUID, 16
|
||||
if name == "Void":
|
||||
return K_UNKNOWN, 0
|
||||
if name in NATIVE_INT:
|
||||
return K_INTEGER, NATIVE_SIZE[name]
|
||||
return K_UNKNOWN, 0
|
||||
|
||||
|
||||
def wants_x64(arches):
|
||||
"""Architectures==[] means arch-neutral; otherwise it must list X64."""
|
||||
return not arches or "X64" in arches
|
||||
|
||||
|
||||
def build_function(md, api, fn, enum_ids):
|
||||
"""Produce (params, flags) for one function, or None if it isn't x64-relevant."""
|
||||
if not wants_x64(fn.get("Architectures") or []):
|
||||
return None
|
||||
|
||||
flags = F_SET_LAST_ERROR if fn.get("SetLastError") else 0
|
||||
|
||||
# A function returning an aggregate that isn't 1/2/4/8 bytes takes a hidden
|
||||
# pointer in RCX, pushing every real parameter one slot to the right.
|
||||
ret = fn.get("ReturnType") or {}
|
||||
ret_node, _ = md.resolve(ret) if ret.get("Kind") == "ApiRef" else (ret, None)
|
||||
base_slot = 0
|
||||
if ret_node.get("Kind") in ("Struct", "Union") or (
|
||||
ret_node.get("Kind") == "Native" and ret_node.get("Name") == "Guid"
|
||||
):
|
||||
size = md.sizeof(ret_node)
|
||||
if size is None:
|
||||
flags |= F_UNSUPPORTED
|
||||
elif size not in (1, 2, 4, 8):
|
||||
flags |= F_HIDDEN_RET_PTR
|
||||
base_slot = 1
|
||||
|
||||
raw_params = fn.get("Params") or []
|
||||
if len(raw_params) + base_slot > MAX_PARAMS:
|
||||
return None
|
||||
|
||||
params = []
|
||||
for i, p in enumerate(raw_params):
|
||||
cp = classify_param(md, p, enum_ids)
|
||||
cp.slot = base_slot + i
|
||||
if cp.kind == K_UNKNOWN:
|
||||
flags |= F_UNSUPPORTED
|
||||
params.append(cp)
|
||||
|
||||
# aux_value indexes into the *parameter* list; the C++ side reads captured
|
||||
# values by parameter index too, so no slot translation is needed. Drop
|
||||
# references that point outside the list rather than trusting them at runtime.
|
||||
for cp in params:
|
||||
if cp.aux_kind in (AUX_BYTES_FROM_PARAM, AUX_COUNT_FROM_PARAM):
|
||||
if not (0 <= cp.aux_value < len(params)):
|
||||
cp.aux_kind, cp.aux_value = AUX_NONE, 0
|
||||
|
||||
return params, flags
|
||||
|
||||
|
||||
def collect_enums(md):
|
||||
"""Index only the enums a parameter can actually reference (950 of 7005)."""
|
||||
referenced = {}
|
||||
|
||||
def visit(t, depth=0):
|
||||
if depth > 16:
|
||||
return
|
||||
kind = t.get("Kind")
|
||||
if kind == "ApiRef":
|
||||
node, _ = md.resolve(t)
|
||||
if node.get("Kind") == "Enum":
|
||||
referenced.setdefault((t.get("Api"), t.get("Name")), node)
|
||||
elif kind in ("PointerTo", "LPArray"):
|
||||
visit(t["Child"], depth + 1)
|
||||
|
||||
for api, fn in md.functions:
|
||||
for p in fn.get("Params") or []:
|
||||
visit(p["Type"])
|
||||
|
||||
enums = []
|
||||
enum_ids = {}
|
||||
for (eapi, ename), node in sorted(referenced.items()):
|
||||
node = dict(node)
|
||||
node["__api__"] = eapi
|
||||
enum_ids[(eapi, ename)] = len(enums)
|
||||
enums.append((eapi, ename, node))
|
||||
return enums, enum_ids
|
||||
|
||||
|
||||
def resolve_enum_index(md, t, enum_ids, depth=0):
|
||||
"""The enum table index a param type refers to, or NO_ENUM."""
|
||||
if depth > 16:
|
||||
return NO_ENUM
|
||||
kind = t.get("Kind")
|
||||
if kind == "ApiRef":
|
||||
node, _ = md.resolve(t)
|
||||
if node.get("Kind") == "Enum":
|
||||
return enum_ids.get((t.get("Api"), t.get("Name")), NO_ENUM)
|
||||
elif kind in ("PointerTo", "LPArray"):
|
||||
return resolve_enum_index(md, t["Child"], enum_ids, depth + 1)
|
||||
return NO_ENUM
|
||||
|
||||
|
||||
def build(api_dir, out_path):
|
||||
print(f"[+] loading {api_dir} ...")
|
||||
md = Metadata(api_dir)
|
||||
print(f"[+] {len(md.functions)} functions, {len(md.types)} types")
|
||||
|
||||
enums, enum_ids = collect_enums(md)
|
||||
print(f"[+] {len(enums)} referenced enums")
|
||||
|
||||
strtab = StringTable()
|
||||
|
||||
# enum tables
|
||||
enum_recs = []
|
||||
enumval_recs = []
|
||||
for eapi, ename, node in enums:
|
||||
values = node.get("Values") or []
|
||||
val_off = len(enumval_recs)
|
||||
for v in values:
|
||||
enumval_recs.append((strtab.add(v["Name"]), int(v["Value"])))
|
||||
enum_recs.append((
|
||||
strtab.add(ename), val_off, len(values),
|
||||
1 if node.get("Flags") else 0, md.enum_width(node),
|
||||
))
|
||||
|
||||
# functions, keyed by bare export name (module is a weak hint only -- API sets
|
||||
# mean kernel32!CreateFileW surfaces as KERNELBASE!CreateFileW in a trace)
|
||||
func_recs = []
|
||||
param_recs = []
|
||||
seen_names = {}
|
||||
skipped_arch = 0
|
||||
unsupported = 0
|
||||
collisions = 0
|
||||
|
||||
for api, fn in sorted(md.functions, key=lambda x: x[1]["Name"]):
|
||||
name = fn["Name"]
|
||||
built = build_function(md, api, fn, enum_ids)
|
||||
if built is None:
|
||||
skipped_arch += 1
|
||||
continue
|
||||
params, flags = built
|
||||
|
||||
if name in seen_names:
|
||||
prior = seen_names[name]
|
||||
if prior != tuple((p.kind, p.slot) for p in params):
|
||||
collisions += 1
|
||||
continue
|
||||
seen_names[name] = tuple((p.kind, p.slot) for p in params)
|
||||
|
||||
for i, p in enumerate(params):
|
||||
enum_idx = resolve_enum_index(md, fn["Params"][i]["Type"], enum_ids)
|
||||
param_recs.append((
|
||||
strtab.add(p.name), strtab.add(p.type_name), p.kind, p.attrs,
|
||||
p.slot, p.aux_kind, p.aux_value, enum_idx, p.pointee_size,
|
||||
))
|
||||
if flags & F_UNSUPPORTED:
|
||||
unsupported += 1
|
||||
func_recs.append((
|
||||
strtab.add(name), strtab.add(fn.get("DllImport") or ""),
|
||||
len(param_recs) - len(params), len(params), flags,
|
||||
))
|
||||
|
||||
print(f"[+] indexed {len(func_recs)} functions "
|
||||
f"({skipped_arch} not x64, {unsupported} flagged UNSUPPORTED, "
|
||||
f"{collisions} name collisions with differing shapes)")
|
||||
print(f"[+] {len(param_recs)} params, {len(enumval_recs)} enum values, "
|
||||
f"{len(strtab.buf)} bytes of strings")
|
||||
|
||||
blob = io.BytesIO()
|
||||
blob.write(MAGIC)
|
||||
blob.write(struct.pack(
|
||||
"<IIIIII", FORMAT_VERSION, len(func_recs), len(param_recs),
|
||||
len(enum_recs), len(enumval_recs), len(strtab.buf),
|
||||
))
|
||||
for name_off, dll_off, param_off, count, flags in func_recs:
|
||||
blob.write(struct.pack("<IIIBBH", name_off, dll_off, param_off, count, flags, 0))
|
||||
for rec in param_recs:
|
||||
name_off, type_off, kind, attrs, slot, aux_kind, aux_value, enum_idx, pointee = rec
|
||||
blob.write(struct.pack("<IIBBBBiIH2x", name_off, type_off, kind, attrs,
|
||||
slot, aux_kind, aux_value, enum_idx, min(pointee, 0xFFFF)))
|
||||
for name_off, val_off, val_count, is_flags, width in enum_recs:
|
||||
blob.write(struct.pack("<IIIBB2x", name_off, val_off, val_count, is_flags, width))
|
||||
for name_off, value in enumval_recs:
|
||||
blob.write(struct.pack("<I4xq", name_off, value))
|
||||
blob.write(strtab.buf)
|
||||
|
||||
out_path = Path(out_path)
|
||||
out_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
out_path.write_bytes(blob.getvalue())
|
||||
print(f"[+] wrote {out_path} ({out_path.stat().st_size / 1024 / 1024:.2f} MB)")
|
||||
return 0
|
||||
|
||||
|
||||
def main(argv):
|
||||
here = Path(__file__).resolve().parent.parent
|
||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[1])
|
||||
ap.add_argument("api_dir", nargs="?", default=str(here / "win32json" / "api"),
|
||||
help="path to win32json/api")
|
||||
ap.add_argument("-o", "--output", default=str(here / "ttd" / "data" / "win32-index.bin"),
|
||||
help="output blob path")
|
||||
args = ap.parse_args(argv)
|
||||
return build(args.api_dir, args.output)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main(sys.argv[1:]))
|
||||
+22
-3
@@ -60,7 +60,19 @@ def main(argv: list[str]) -> int:
|
||||
parser.add_argument("--sample", help="optional on-disk sample for accurate hashes")
|
||||
parser.add_argument("--extractor", help="path to ttdcapa-extract.exe")
|
||||
parser.add_argument("--max-calls", type=int, help="cap recorded API calls (for huge traces)")
|
||||
parser.add_argument("--with-stack-args", action="store_true", help="also capture stack args 5+")
|
||||
parser.add_argument(
|
||||
"--with-stack-args",
|
||||
action="store_true",
|
||||
help="for calls with no Win32 metadata, also grab four stack slots past the "
|
||||
"register args (calls we have a signature for always capture their true arity)",
|
||||
)
|
||||
parser.add_argument("--win32-index", help="path to win32-index.bin (default: next to the extractor)")
|
||||
parser.add_argument(
|
||||
"--no-metadata",
|
||||
action="store_true",
|
||||
help="disable metadata-driven argument decoding entirely",
|
||||
)
|
||||
parser.add_argument("--max-buffer", type=int, help="bytes kept from any one captured buffer (default 256)")
|
||||
parser.add_argument("--keep-json", action="store_true", help="keep the intermediate TTD report")
|
||||
parser.add_argument("--python", default=sys.executable, help="python interpreter to run capa")
|
||||
args, capa_extra = parser.parse_known_args(argv)
|
||||
@@ -71,9 +83,10 @@ def main(argv: list[str]) -> int:
|
||||
trace = Path(args.trace)
|
||||
if not trace.is_file():
|
||||
sys.exit(f"trace not found: {trace}")
|
||||
rules = Path(args.rules)
|
||||
# absolute, because capa is invoked with cwd set to the report's directory
|
||||
rules = Path(args.rules).resolve()
|
||||
if not rules.exists():
|
||||
sys.exit(f"rules path not found: {rules}")
|
||||
sys.exit(f"rules path not found: {args.rules}")
|
||||
|
||||
extractor = find_extractor(args.extractor)
|
||||
|
||||
@@ -89,6 +102,12 @@ def main(argv: list[str]) -> int:
|
||||
extract_cmd += ["--max-calls", str(args.max_calls)]
|
||||
if args.with_stack_args:
|
||||
extract_cmd += ["--with-stack-args"]
|
||||
if args.win32_index:
|
||||
extract_cmd += ["--win32-index", args.win32_index]
|
||||
if args.no_metadata:
|
||||
extract_cmd += ["--no-metadata"]
|
||||
if args.max_buffer:
|
||||
extract_cmd += ["--max-buffer", str(args.max_buffer)]
|
||||
|
||||
print(f"[ttd-capa] extracting: {' '.join(extract_cmd)}", file=sys.stderr)
|
||||
rc = subprocess.call(extract_cmd)
|
||||
|
||||
@@ -28,6 +28,20 @@ from typing import Optional
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
|
||||
# Recovered guest strings are arbitrary text -- non-Latin paths, HTTP bodies, the
|
||||
# occasional lone surrogate from a half-written UTF-16 buffer. When stdout is a
|
||||
# console Python uses UTF-8, but when it is redirected to a file it falls back to
|
||||
# the locale encoding (cp1252 here), which cannot encode any of that and aborts the
|
||||
# whole run. Pin UTF-8 and degrade unencodable characters instead of dying.
|
||||
for _stream in (sys.stdout, sys.stderr):
|
||||
if hasattr(_stream, "reconfigure"):
|
||||
_stream.reconfigure(encoding="utf-8", errors="replace")
|
||||
|
||||
# Python puts this script's directory first on sys.path, and the capa fork lives in
|
||||
# a `capa/` subdirectory of it -- which shadows the installed `capa` package with a
|
||||
# namespace package that has no submodules. Drop it; nothing here is imported locally.
|
||||
sys.path[:] = [p for p in sys.path if p and Path(p).resolve() != HERE]
|
||||
|
||||
from capa.features.address import ProcessAddress, ThreadAddress, DynamicCallAddress # noqa: E402
|
||||
from capa.features.extractors.ttd.models import TtdCall # noqa: E402
|
||||
from capa.features.extractors.ttd.extractor import TtdExtractor # noqa: E402
|
||||
@@ -66,8 +80,32 @@ def fmt_arg(a) -> str:
|
||||
return repr(a)
|
||||
|
||||
|
||||
def fmt_param(p) -> str:
|
||||
"""One metadata-decoded parameter as `name=value`.
|
||||
|
||||
Prefers the most informative rendering the extractor managed: decoded text,
|
||||
then symbolic flag names, then a dereferenced pointee, then the raw value.
|
||||
"""
|
||||
if p.str_ is not None:
|
||||
value = repr(p.str_)
|
||||
elif p.flags:
|
||||
value = "|".join(p.flags)
|
||||
elif p.float_ is not None:
|
||||
value = repr(p.float_)
|
||||
else:
|
||||
value = fmt_arg(p.value)
|
||||
if p.deref is not None:
|
||||
value += f"->{fmt_arg(p.deref)}"
|
||||
if p.at_return:
|
||||
value += "@ret"
|
||||
return f"{p.name}={value}" if p.name else value
|
||||
|
||||
|
||||
def format_call(call: TtdCall) -> str:
|
||||
api = f"{call.module}.{call.api}" if call.module else call.api
|
||||
if call.params:
|
||||
args = ", ".join(fmt_param(p) for p in call.params)
|
||||
else:
|
||||
args = ", ".join(fmt_arg(a) for a in call.args)
|
||||
ret = "" if call.ret is None else f" -> 0x{call.ret & 0xFFFFFFFFFFFFFFFF:x}"
|
||||
return f"{api}({args}){ret}"
|
||||
|
||||
Binary file not shown.
@@ -0,0 +1,422 @@
|
||||
#include "abi_x64.hpp"
|
||||
|
||||
#include <cstdio>
|
||||
#include <cstring>
|
||||
|
||||
using ttdcapa::win32meta::ArgKind;
|
||||
using ttdcapa::win32meta::AuxKind;
|
||||
|
||||
namespace ttdcapa {
|
||||
namespace {
|
||||
|
||||
// A dereference is only worth attempting above the first 64 KiB; the null
|
||||
// page and its neighbours are never mapped in user mode.
|
||||
constexpr uint64_t kMinDerefAddr = 0x10000;
|
||||
|
||||
// Guards against a mis-typed count parameter turning into a huge read.
|
||||
constexpr uint64_t kMaxCountElements = 1u << 20;
|
||||
|
||||
bool readGuest(TTD::Replay::IThreadView const* thread, uint64_t addr, void* dst, size_t size) {
|
||||
if (addr < kMinDerefAddr || size == 0) {
|
||||
return false;
|
||||
}
|
||||
auto result = thread->QueryMemoryBuffer(TTD::GuestAddress{ addr }, TTD::BufferView{ dst, size });
|
||||
return result.Memory.Size == size;
|
||||
}
|
||||
|
||||
// The value in `slot`, honouring the shared integer/SSE slot numbering.
|
||||
uint64_t fetchSlot(const AMD64_CONTEXT& ctx, TTD::Replay::IThreadView const* thread,
|
||||
uint8_t slot, bool isFloat, bool& ok) {
|
||||
ok = true;
|
||||
if (slot < 4) {
|
||||
if (isFloat) {
|
||||
const M128BIT* xmm[4] = { &ctx.Xmm0, &ctx.Xmm1, &ctx.Xmm2, &ctx.Xmm3 };
|
||||
return xmm[slot]->Low;
|
||||
}
|
||||
const uint64_t gpr[4] = { ctx.Rcx, ctx.Rdx, ctx.R8, ctx.R9 };
|
||||
return gpr[slot];
|
||||
}
|
||||
// Above the shadow space the caller reserved for RCX/RDX/R8/R9.
|
||||
uint64_t addr = ctx.Rsp + 0x28 + static_cast<uint64_t>(slot - 4) * 8;
|
||||
uint64_t v = 0;
|
||||
ok = readGuest(thread, addr, &v, sizeof(v));
|
||||
return v;
|
||||
}
|
||||
|
||||
double floatValue(const win32meta::ParamSig& p, uint64_t bits) {
|
||||
if (p.kind == ArgKind::Float) {
|
||||
float f = 0.0f;
|
||||
uint32_t lo = static_cast<uint32_t>(bits);
|
||||
std::memcpy(&f, &lo, sizeof(f));
|
||||
return static_cast<double>(f);
|
||||
}
|
||||
double d = 0.0;
|
||||
std::memcpy(&d, &bits, sizeof(d));
|
||||
return d;
|
||||
}
|
||||
|
||||
// Widen a pointee of `size` bytes to 64 bits.
|
||||
uint64_t narrowRead(const uint8_t* raw, uint16_t size) {
|
||||
uint64_t v = 0;
|
||||
std::memcpy(&v, raw, size > 8 ? 8 : size);
|
||||
return v;
|
||||
}
|
||||
|
||||
bool derefScalar(TTD::Replay::IThreadView const* thread, uint64_t ptr, uint16_t size, uint64_t& out) {
|
||||
uint16_t width = size == 0 ? 8 : (size > 8 ? 8 : size);
|
||||
uint8_t buf[8] = {};
|
||||
if (!readGuest(thread, ptr, buf, width)) {
|
||||
return false;
|
||||
}
|
||||
out = narrowRead(buf, width);
|
||||
return true;
|
||||
}
|
||||
|
||||
std::string formatGuid(const uint8_t* b) {
|
||||
char buf[40];
|
||||
std::snprintf(buf, sizeof(buf),
|
||||
"{%08lX-%04X-%04X-%02X%02X-%02X%02X%02X%02X%02X%02X}",
|
||||
static_cast<unsigned long>(narrowRead(b, 4)),
|
||||
static_cast<unsigned>(narrowRead(b + 4, 2)),
|
||||
static_cast<unsigned>(narrowRead(b + 6, 2)),
|
||||
b[8], b[9], b[10], b[11], b[12], b[13], b[14], b[15]);
|
||||
return buf;
|
||||
}
|
||||
|
||||
bool isBufferKind(ArgKind kind) {
|
||||
return kind == ArgKind::AnsiBuffer || kind == ArgKind::WideBuffer || kind == ArgKind::ByteBuffer;
|
||||
}
|
||||
|
||||
// How many bytes a counted buffer parameter spans, or 0 when we can't tell.
|
||||
// `args` supplies the sibling parameter the count lives in -- after a return
|
||||
// pass those may themselves have been filled in, which is exactly what makes
|
||||
// ReadFile's lpBuffer renderable at its *actual* length.
|
||||
uint64_t resolveByteCount(const win32meta::ParamSig& p, const std::vector<DecodedArg>& args) {
|
||||
uint64_t count = 0;
|
||||
switch (p.auxKind) {
|
||||
case AuxKind::CountConst:
|
||||
count = static_cast<uint64_t>(p.auxValue);
|
||||
break;
|
||||
case AuxKind::BytesFromParam:
|
||||
case AuxKind::CountFromParam: {
|
||||
if (p.auxValue < 0 || static_cast<size_t>(p.auxValue) >= args.size()) {
|
||||
return 0;
|
||||
}
|
||||
const DecodedArg& src = args[static_cast<size_t>(p.auxValue)];
|
||||
count = src.has_deref ? src.deref : src.raw;
|
||||
break;
|
||||
}
|
||||
case AuxKind::None:
|
||||
default:
|
||||
return 0;
|
||||
}
|
||||
if (count == 0 || count > kMaxCountElements) {
|
||||
return 0;
|
||||
}
|
||||
if (p.auxKind == AuxKind::BytesFromParam) {
|
||||
return count;
|
||||
}
|
||||
uint16_t elem = p.pointeeSize ? p.pointeeSize : 1;
|
||||
return count * elem;
|
||||
}
|
||||
|
||||
// Length of `buf` up to and including its first `charWidth`-wide NUL. A
|
||||
// character buffer's count parameter is the caller's *capacity*, so without
|
||||
// this the report would carry a few hundred bytes of unrelated stack memory
|
||||
// after every out-string.
|
||||
size_t terminatorEnd(const std::vector<uint8_t>& buf, size_t charWidth) {
|
||||
for (size_t i = 0; i + charWidth <= buf.size(); i += charWidth) {
|
||||
bool nul = true;
|
||||
for (size_t k = 0; k < charWidth; ++k) {
|
||||
if (buf[i + k] != 0) {
|
||||
nul = false;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (nul) {
|
||||
return i + charWidth;
|
||||
}
|
||||
}
|
||||
return buf.size();
|
||||
}
|
||||
|
||||
// Read a counted buffer into `arg`, capped at opt.max_buffer. Character
|
||||
// buffers additionally get a textual rendering, since that's what a rule or
|
||||
// an analyst actually wants to see.
|
||||
void captureBuffer(TTD::Replay::IThreadView const* thread, const DecodeOptions& opt,
|
||||
ArgKind kind, uint64_t ptr, uint64_t byteCount, DecodedArg& arg) {
|
||||
if (ptr < kMinDerefAddr || byteCount == 0) {
|
||||
return;
|
||||
}
|
||||
size_t want = static_cast<size_t>(byteCount < opt.max_buffer ? byteCount : opt.max_buffer);
|
||||
std::vector<uint8_t> buf(want);
|
||||
auto result = thread->QueryMemoryBuffer(TTD::GuestAddress{ ptr }, TTD::BufferView{ buf.data(), want });
|
||||
size_t got = result.Memory.Size;
|
||||
if (got == 0) {
|
||||
return;
|
||||
}
|
||||
buf.resize(got);
|
||||
|
||||
if (kind == ArgKind::AnsiBuffer) {
|
||||
if (auto s = readAnsiString(thread, ptr, got)) {
|
||||
arg.str = std::move(*s);
|
||||
arg.has_str = !arg.str.empty();
|
||||
}
|
||||
buf.resize(terminatorEnd(buf, 1));
|
||||
} else if (kind == ArgKind::WideBuffer) {
|
||||
if (auto s = readWideString(thread, ptr, got / sizeof(wchar_t))) {
|
||||
arg.str = std::move(*s);
|
||||
arg.has_str = !arg.str.empty();
|
||||
}
|
||||
buf.resize(terminatorEnd(buf, 2));
|
||||
}
|
||||
arg.bytes = std::move(buf);
|
||||
}
|
||||
|
||||
// Everything that needs the callee to have run. Shared by the entry pass
|
||||
// (for [In] parameters, which are already valid) and the return pass.
|
||||
void dereference(TTD::Replay::IThreadView const* thread, const DecodeOptions& opt,
|
||||
ArgKind kind, uint64_t ptr, uint16_t pointeeSize,
|
||||
uint64_t byteCount, DecodedArg& arg) {
|
||||
switch (kind) {
|
||||
case ArgKind::AnsiString:
|
||||
if (auto s = readAnsiString(thread, ptr, opt.max_string)) {
|
||||
arg.str = std::move(*s);
|
||||
arg.has_str = true;
|
||||
}
|
||||
break;
|
||||
case ArgKind::WideString:
|
||||
if (auto s = readWideString(thread, ptr, opt.max_string)) {
|
||||
arg.str = std::move(*s);
|
||||
arg.has_str = true;
|
||||
}
|
||||
break;
|
||||
case ArgKind::PtrToInt: {
|
||||
uint64_t v = 0;
|
||||
if (derefScalar(thread, ptr, pointeeSize, v)) {
|
||||
arg.deref = v;
|
||||
arg.has_deref = true;
|
||||
}
|
||||
break;
|
||||
}
|
||||
case ArgKind::PtrToAnsiString:
|
||||
case ArgKind::PtrToWideString: {
|
||||
uint64_t inner = 0;
|
||||
if (!derefScalar(thread, ptr, 8, inner)) {
|
||||
break;
|
||||
}
|
||||
arg.deref = inner;
|
||||
arg.has_deref = true;
|
||||
auto s = (kind == ArgKind::PtrToAnsiString)
|
||||
? readAnsiString(thread, inner, opt.max_string)
|
||||
: readWideString(thread, inner, opt.max_string);
|
||||
if (s) {
|
||||
arg.str = std::move(*s);
|
||||
arg.has_str = true;
|
||||
}
|
||||
break;
|
||||
}
|
||||
case ArgKind::Guid: {
|
||||
uint8_t g[16] = {};
|
||||
if (readGuest(thread, ptr, g, sizeof(g))) {
|
||||
arg.str = formatGuid(g);
|
||||
arg.has_str = true;
|
||||
}
|
||||
break;
|
||||
}
|
||||
case ArgKind::AnsiBuffer:
|
||||
case ArgKind::WideBuffer:
|
||||
case ArgKind::ByteBuffer:
|
||||
captureBuffer(thread, opt, kind, ptr, byteCount, arg);
|
||||
break;
|
||||
default:
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
// Some parameters are pointers whose pointee the metadata can't pin down:
|
||||
// opaque void*, and the raw UInt16*/UIntPtr* that RPC uses for RPC_WSTR.
|
||||
// For those the old guess-if-it-looks-like-text heuristic is still the best
|
||||
// information available -- and unlike before, we now only apply it to values
|
||||
// we know really are parameters and really are pointers.
|
||||
bool mayHoldUntypedString(ArgKind kind) {
|
||||
return kind == ArgKind::Pointer || kind == ArgKind::Unknown || kind == ArgKind::PtrToInt;
|
||||
}
|
||||
|
||||
void tryUntypedString(TTD::Replay::IThreadView const* thread, DecodedArg& arg) {
|
||||
if (arg.has_str || !mayHoldUntypedString(arg.kind)) {
|
||||
return;
|
||||
}
|
||||
if (auto s = tryReadString(thread, arg.raw)) {
|
||||
arg.str = std::move(*s);
|
||||
arg.has_str = true;
|
||||
return;
|
||||
}
|
||||
// A T** out-parameter: the string lives one more hop away.
|
||||
if (arg.has_deref) {
|
||||
if (auto s = tryReadString(thread, arg.deref)) {
|
||||
arg.str = std::move(*s);
|
||||
arg.has_str = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
bool needsDeref(ArgKind kind) {
|
||||
switch (kind) {
|
||||
case ArgKind::AnsiString:
|
||||
case ArgKind::WideString:
|
||||
case ArgKind::PtrToInt:
|
||||
case ArgKind::PtrToAnsiString:
|
||||
case ArgKind::PtrToWideString:
|
||||
case ArgKind::Guid:
|
||||
case ArgKind::AnsiBuffer:
|
||||
case ArgKind::WideBuffer:
|
||||
case ArgKind::ByteBuffer:
|
||||
return true;
|
||||
default:
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
void decodeArgs(const win32meta::FuncSig& sig,
|
||||
const AMD64_CONTEXT& ctx,
|
||||
TTD::Replay::IThreadView const* thread,
|
||||
const DecodeOptions& opt,
|
||||
std::vector<DecodedArg>& out,
|
||||
std::vector<PendingOut>& deferred) {
|
||||
out.clear();
|
||||
out.resize(sig.paramCount);
|
||||
|
||||
// Pass 1: capture every raw slot first. A buffer's length can live in a
|
||||
// parameter that comes *after* it (ReadFile's lpBuffer refers forward to
|
||||
// nNumberOfBytesToRead), so no dereferencing until all the scalars are in.
|
||||
for (uint8_t i = 0; i < sig.paramCount; ++i) {
|
||||
const win32meta::ParamSig& p = sig.params[i];
|
||||
DecodedArg& arg = out[i];
|
||||
arg.name = p.name;
|
||||
arg.type = p.type;
|
||||
arg.kind = p.kind;
|
||||
arg.enum_index = p.enumIndex;
|
||||
arg.is_out = p.isOut();
|
||||
|
||||
bool ok = false;
|
||||
arg.raw = fetchSlot(ctx, thread, p.slot, p.isFloat(), ok);
|
||||
if (!ok) {
|
||||
// Stack slot we couldn't read: leave it zero rather than invent one.
|
||||
arg.raw = 0;
|
||||
}
|
||||
if (p.isFloat()) {
|
||||
arg.fval = floatValue(p, arg.raw);
|
||||
arg.has_fval = true;
|
||||
}
|
||||
}
|
||||
|
||||
// Pass 2: dereference. Scalars and handles are deliberately untouched --
|
||||
// that alone removes most of the bogus String features the old heuristic
|
||||
// produced from flag values that happened to look like addresses.
|
||||
for (uint8_t i = 0; i < sig.paramCount; ++i) {
|
||||
const win32meta::ParamSig& p = sig.params[i];
|
||||
DecodedArg& arg = out[i];
|
||||
if (!needsDeref(p.kind)) {
|
||||
tryUntypedString(thread, arg);
|
||||
continue;
|
||||
}
|
||||
|
||||
uint64_t byteCount = isBufferKind(p.kind) ? resolveByteCount(p, out) : 0;
|
||||
|
||||
// [In] contents are already valid here. [In,Out] gets read twice: once
|
||||
// now for what the caller passed, then again at the return.
|
||||
if (p.isIn()) {
|
||||
dereference(thread, opt, p.kind, arg.raw, p.pointeeSize, byteCount, arg);
|
||||
tryUntypedString(thread, arg);
|
||||
}
|
||||
if (p.isOut() && arg.raw >= kMinDerefAddr) {
|
||||
PendingOut pending;
|
||||
pending.param_index = i;
|
||||
pending.kind = p.kind;
|
||||
pending.ptr = arg.raw;
|
||||
pending.pointee_size = p.pointeeSize;
|
||||
pending.aux_kind = p.auxKind;
|
||||
pending.aux_value = p.auxValue;
|
||||
pending.in_cap = byteCount;
|
||||
deferred.push_back(pending);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
void resolvePendingOuts(const std::vector<PendingOut>& pending,
|
||||
TTD::Replay::IThreadView const* thread,
|
||||
const DecodeOptions& opt,
|
||||
std::vector<DecodedArg>& args) {
|
||||
// Scalars first: a buffer's real length is usually itself an [Out] scalar
|
||||
// (ReadFile's lpNumberOfBytesRead), so it has to be resolved before the
|
||||
// buffer that depends on it.
|
||||
for (const PendingOut& po : pending) {
|
||||
if (po.param_index >= args.size() || isBufferKind(po.kind)) {
|
||||
continue;
|
||||
}
|
||||
DecodedArg& arg = args[po.param_index];
|
||||
DecodedArg fresh;
|
||||
dereference(thread, opt, po.kind, po.ptr, po.pointee_size, 0, fresh);
|
||||
if (fresh.has_deref || fresh.has_str) {
|
||||
fresh.name = arg.name;
|
||||
fresh.type = arg.type;
|
||||
fresh.kind = arg.kind;
|
||||
fresh.enum_index = arg.enum_index;
|
||||
fresh.raw = arg.raw;
|
||||
fresh.is_out = true;
|
||||
fresh.from_return = true;
|
||||
arg = std::move(fresh);
|
||||
}
|
||||
tryUntypedString(thread, arg);
|
||||
}
|
||||
|
||||
for (const PendingOut& po : pending) {
|
||||
if (po.param_index >= args.size() || !isBufferKind(po.kind)) {
|
||||
continue;
|
||||
}
|
||||
DecodedArg& arg = args[po.param_index];
|
||||
|
||||
// Prefer the length the callee reported; fall back to what the caller
|
||||
// offered, and never exceed it.
|
||||
win32meta::ParamSig probe;
|
||||
probe.auxKind = po.aux_kind;
|
||||
probe.auxValue = po.aux_value;
|
||||
probe.pointeeSize = po.pointee_size;
|
||||
uint64_t byteCount = resolveByteCount(probe, args);
|
||||
if (byteCount == 0) {
|
||||
byteCount = po.in_cap;
|
||||
} else if (po.in_cap != 0 && byteCount > po.in_cap) {
|
||||
byteCount = po.in_cap;
|
||||
}
|
||||
if (byteCount == 0) {
|
||||
continue;
|
||||
}
|
||||
|
||||
DecodedArg fresh;
|
||||
captureBuffer(thread, opt, po.kind, po.ptr, byteCount, fresh);
|
||||
if (!fresh.bytes.empty() || fresh.has_str) {
|
||||
arg.bytes = std::move(fresh.bytes);
|
||||
arg.str = std::move(fresh.str);
|
||||
arg.has_str = fresh.has_str;
|
||||
arg.from_return = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
std::vector<ArgValue> toCapaArgs(const std::vector<DecodedArg>& args) {
|
||||
std::vector<ArgValue> out;
|
||||
out.reserve(args.size());
|
||||
for (const DecodedArg& a : args) {
|
||||
if (a.has_str && !a.str.empty()) {
|
||||
out.push_back(a.str);
|
||||
} else {
|
||||
out.push_back(static_cast<int64_t>(a.raw));
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
} // namespace ttdcapa
|
||||
@@ -0,0 +1,60 @@
|
||||
#ifndef ABI_X64_HPP
|
||||
#define ABI_X64_HPP
|
||||
|
||||
// The x64 calling-convention decoder.
|
||||
//
|
||||
// The Win32 metadata gives us types and semantics; mapping those onto registers
|
||||
// and stack slots is ours to write (WIN32JSON-TTD-INTEGRATION-NOTES.md section 3).
|
||||
// That's this file: given a signature and a thread's register state at a CALL,
|
||||
// produce one DecodedArg per real parameter -- no more, no less.
|
||||
//
|
||||
// Microsoft x64 in one paragraph: the first four parameters go in RCX/RDX/R8/R9,
|
||||
// or XMM0-3 if they're floating point. Crucially the slot index is *shared* between
|
||||
// the two register files, so a float in position 2 lives in XMM2, not XMM0.
|
||||
// Parameters five and up sit at [RSP+0x28] onwards, above the 32-byte shadow space
|
||||
// the caller must reserve. Aggregates that aren't exactly 1/2/4/8 bytes are passed
|
||||
// by hidden pointer, and a function returning one takes an extra hidden first
|
||||
// parameter -- the indexer pre-computes both, so `slot` here is always final.
|
||||
|
||||
#include <cstdint>
|
||||
#include <vector>
|
||||
|
||||
#include <TTD/IReplayEngineStl.h>
|
||||
#include <TTD/IReplayEngineRegisters.h>
|
||||
|
||||
#include "ttdutils.hpp"
|
||||
#include "win32meta.hpp"
|
||||
|
||||
namespace ttdcapa {
|
||||
|
||||
struct DecodeOptions {
|
||||
size_t max_buffer = 256; // bytes of any one counted buffer to keep
|
||||
size_t max_string = 512; // characters of any one string to keep
|
||||
};
|
||||
|
||||
// Decode every parameter of `sig` from the register state at the call.
|
||||
// Dereferences that only make sense once the callee has run are appended to
|
||||
// `deferred` instead; feed those to resolvePendingOuts at the return.
|
||||
void decodeArgs(const win32meta::FuncSig& sig,
|
||||
const AMD64_CONTEXT& ctx,
|
||||
TTD::Replay::IThreadView const* thread,
|
||||
const DecodeOptions& opt,
|
||||
std::vector<DecodedArg>& out,
|
||||
std::vector<PendingOut>& deferred);
|
||||
|
||||
// Re-read the deferred dereferences at the return position and patch them into
|
||||
// `args` (which must be the vector decodeArgs filled in for this same call).
|
||||
void resolvePendingOuts(const std::vector<PendingOut>& pending,
|
||||
TTD::Replay::IThreadView const* thread,
|
||||
const DecodeOptions& opt,
|
||||
std::vector<DecodedArg>& args);
|
||||
|
||||
// Flatten decoded parameters into the int/string list capa matches rules
|
||||
// against. Strings recovered from [Out] parameters are included -- they are
|
||||
// genuine new evidence; symbolic enum names are not, so rules that match flags
|
||||
// numerically keep working.
|
||||
std::vector<ArgValue> toCapaArgs(const std::vector<DecodedArg>& args);
|
||||
|
||||
} // namespace ttdcapa
|
||||
|
||||
#endif
|
||||
+99
-12
@@ -3,7 +3,13 @@
|
||||
// (capa/features/extractors/ttd/). x64 traces only in v1.
|
||||
//
|
||||
// ttdcapa-extract <trace.run> [--sample <sample.exe>] [-o <out.json>]
|
||||
// [--max-calls N] [--with-stack-args]
|
||||
// [--max-calls N] [--with-stack-args] [--win32-index <path>]
|
||||
// [--no-metadata] [--max-buffer N]
|
||||
// ttdcapa-extract --dump-sig <ApiName>
|
||||
//
|
||||
// Arguments are decoded against the pre-baked Win32 metadata index whenever the
|
||||
// resolved export is in it (see win32meta.hpp / abi_x64.hpp); calls we have no
|
||||
// signature for fall back to the original heuristic capture.
|
||||
|
||||
#include <cassert>
|
||||
#define DBG_ASSERT(cond) assert(cond)
|
||||
@@ -30,18 +36,67 @@
|
||||
#include "ttdutils.hpp"
|
||||
#include "utils.hpp"
|
||||
#include "ttd_pe_utils.hpp"
|
||||
#include "win32meta.hpp"
|
||||
#include "abi_x64.hpp"
|
||||
|
||||
using namespace ttdcapa;
|
||||
|
||||
Report g_report;
|
||||
|
||||
// Print one function's decoded signature and exit. Lets the metadata index and the
|
||||
// ABI classification be checked without waiting on a trace replay.
|
||||
static int dumpSignature(const std::string& api) {
|
||||
const win32meta::FuncSig* sig = win32meta::index().lookup(api);
|
||||
if (sig == nullptr) {
|
||||
std::cerr << "[-] '" << api << "' is not in the metadata index\n";
|
||||
return 3;
|
||||
}
|
||||
std::cout << sig->name << " dll=" << sig->dll
|
||||
<< " params=" << static_cast<int>(sig->paramCount)
|
||||
<< (sig->hiddenRetPtr() ? " [hidden-return-pointer]" : "")
|
||||
<< (sig->unsupported() ? " [unsupported]" : "") << "\n";
|
||||
for (uint8_t i = 0; i < sig->paramCount; ++i) {
|
||||
const win32meta::ParamSig& p = sig->params[i];
|
||||
std::cout << " slot " << static_cast<int>(p.slot) << " " << p.name
|
||||
<< " : " << p.type << " (" << win32meta::kindName(p.kind) << ")";
|
||||
if (p.isIn()) std::cout << " in";
|
||||
if (p.isOut()) std::cout << " out";
|
||||
if (p.attrs & win32meta::AttrOptional) std::cout << " optional";
|
||||
if (p.auxKind == win32meta::AuxKind::BytesFromParam) std::cout << " bytes=param[" << p.auxValue << "]";
|
||||
if (p.auxKind == win32meta::AuxKind::CountFromParam) std::cout << " count=param[" << p.auxValue << "]";
|
||||
if (p.auxKind == win32meta::AuxKind::CountConst) std::cout << " count=" << p.auxValue;
|
||||
if (p.hasEnum()) std::cout << " enum=" << win32meta::index().enumName(p.enumIndex);
|
||||
std::cout << "\n";
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
int wmain(int argc, wchar_t** argv) {
|
||||
Options opt;
|
||||
if (!parse_args(argc, argv, opt)) {
|
||||
std::cerr << "Usage: ttdcapa-extract <trace.run> [--sample <sample.exe>] [-o <out.json>] [--max-calls N] [--with-stack-args]\n";
|
||||
std::cerr << "Usage: ttdcapa-extract <trace.run> [--sample <sample.exe>] [-o <out.json>]\n"
|
||||
" [--max-calls N] [--with-stack-args]\n"
|
||||
" [--win32-index <path>] [--no-metadata] [--max-buffer N]\n"
|
||||
" ttdcapa-extract --dump-sig <ApiName>\n";
|
||||
return 1;
|
||||
}
|
||||
|
||||
DecodeOptions decode_opt;
|
||||
decode_opt.max_buffer = opt.max_buffer;
|
||||
if (!opt.no_metadata) {
|
||||
std::string err;
|
||||
if (win32meta::loadIndex(opt.win32_index, err)) {
|
||||
std::cerr << "[+] Loaded Win32 metadata for " << win32meta::index().functionCount()
|
||||
<< " functions\n";
|
||||
} else {
|
||||
std::cerr << "[!] " << err << "; falling back to heuristic argument capture\n";
|
||||
}
|
||||
}
|
||||
|
||||
if (!opt.dump_sig.empty()) {
|
||||
return dumpSignature(opt.dump_sig);
|
||||
}
|
||||
|
||||
auto [engine, hr] = TTD::Replay::MakeReplayEngine();
|
||||
if (hr != S_OK || !engine) {
|
||||
std::cerr << "[-] Failed to create replay engine: 0x" << std::hex << hr << "\n";
|
||||
@@ -71,7 +126,8 @@ int wmain(int argc, wchar_t** argv) {
|
||||
// Get list of function VAs used for call sweep
|
||||
std::unordered_map<uint64_t, std::pair<std::wstring, std::string>> resolvedTraceModuleExports;
|
||||
resolvedTraceModuleExports = resolveTraceModuleExports(engine, inspection_cursor);
|
||||
std::cerr << "[+] Resolved all exported module functions across execution\n";
|
||||
std::cerr << "[+] Resolved " << resolvedTraceModuleExports.size()
|
||||
<< " exported module functions across execution\n";
|
||||
|
||||
size_t thread_count = engine->GetThreadCount();
|
||||
TTD::Replay::ThreadInfo const* thread_list = engine->GetThreadList();
|
||||
@@ -79,9 +135,16 @@ int wmain(int argc, wchar_t** argv) {
|
||||
g_report.process.threads.push_back(static_cast<uint64_t>(thread_list[i].UniqueId));
|
||||
}
|
||||
|
||||
// per-thread stack of in-flight recorded calls: (expected return addr, call index)
|
||||
std::unordered_map<uint64_t, std::vector<std::pair<uint64_t, size_t>>> in_flight;
|
||||
// per-thread stack of recorded calls we haven't seen return yet, each carrying
|
||||
// the [Out] dereferences we deliberately postponed until the callee has run
|
||||
struct InFlight {
|
||||
uint64_t ret_addr = 0;
|
||||
size_t call_index = 0;
|
||||
std::vector<PendingOut> pending;
|
||||
};
|
||||
std::unordered_map<uint64_t, std::vector<InFlight>> in_flight;
|
||||
uint64_t seq = 0;
|
||||
uint64_t events_seen = 0;
|
||||
bool limit_hit = false;
|
||||
|
||||
TTD::Replay::UniqueCursor sweep{ engine->NewCursor() };
|
||||
@@ -90,6 +153,7 @@ int wmain(int argc, wchar_t** argv) {
|
||||
TTD::Replay::IThreadView const* thread) noexcept {
|
||||
bool is_call = (static_cast<uint64_t>(fall_through) != 0);
|
||||
uint64_t utid = static_cast<uint64_t>(thread->GetThreadInfo().UniqueId);
|
||||
++events_seen;
|
||||
|
||||
if (is_call) {
|
||||
auto it = resolvedTraceModuleExports.find(static_cast<uint64_t>(target));
|
||||
@@ -120,6 +184,16 @@ int wmain(int argc, wchar_t** argv) {
|
||||
|
||||
TTD::Replay::RegisterContext regs = thread->GetCrossPlatformContext();
|
||||
auto const* ctx = reinterpret_cast<AMD64_CONTEXT const*>(®s);
|
||||
|
||||
std::vector<PendingOut> pending;
|
||||
win32meta::FuncSig const* sig = win32meta::index().lookup(rec.api);
|
||||
if (sig != nullptr && !sig->unsupported()) {
|
||||
// We know the real arity, so capture exactly that many arguments
|
||||
// -- no stale RDX/R8/R9 residue masquerading as parameters.
|
||||
decodeArgs(*sig, *ctx, thread, decode_opt, rec.params, pending);
|
||||
rec.args = toCapaArgs(rec.params);
|
||||
rec.metadata = true;
|
||||
} else {
|
||||
rec.args.push_back(captureCallArg(thread, ctx->Rcx));
|
||||
rec.args.push_back(captureCallArg(thread, ctx->Rdx));
|
||||
rec.args.push_back(captureCallArg(thread, ctx->R8));
|
||||
@@ -135,18 +209,26 @@ int wmain(int argc, wchar_t** argv) {
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
size_t idx = g_report.process.calls.size();
|
||||
in_flight[utid].push_back({ static_cast<uint64_t>(fall_through), idx });
|
||||
in_flight[utid].push_back(InFlight{ static_cast<uint64_t>(fall_through), idx, std::move(pending) });
|
||||
g_report.process.calls.push_back(std::move(rec));
|
||||
} else {
|
||||
// Log most recent function called as returned and capture return value
|
||||
auto it = in_flight.find(utid);
|
||||
if (it != in_flight.end() && !it->second.empty()) {
|
||||
auto& top = it->second.back();
|
||||
if (top.first == static_cast<uint64_t>(target)) {
|
||||
g_report.process.calls[top.second].ret = thread->GetBasicReturnValue();
|
||||
g_report.process.calls[top.second].has_ret = true;
|
||||
InFlight& top = it->second.back();
|
||||
if (top.ret_addr == static_cast<uint64_t>(target)) {
|
||||
CallRecord& call = g_report.process.calls[top.call_index];
|
||||
call.ret = thread->GetBasicReturnValue();
|
||||
call.has_ret = true;
|
||||
if (!top.pending.empty()) {
|
||||
// The payoff of a time-travel trace: [Out] parameters can be
|
||||
// rendered filled in, which a live debugger can't easily do.
|
||||
resolvePendingOuts(top.pending, thread, decode_opt, call.params);
|
||||
call.args = toCapaArgs(call.params);
|
||||
}
|
||||
it->second.pop_back();
|
||||
}
|
||||
}
|
||||
@@ -154,7 +236,11 @@ int wmain(int argc, wchar_t** argv) {
|
||||
};
|
||||
|
||||
sweep->SetCallReturnCallback(on_call_return);
|
||||
sweep->SetReplayFlags(TTD::Replay::ReplayFlags::ReplaySegmentsSequentially);
|
||||
// Without ReplayAllSegmentsWithoutFiltering the engine only replays segments it
|
||||
// thinks can hit an event, and a call/return callback alone doesn't qualify --
|
||||
// the sweep then completes instantly having seen nothing.
|
||||
sweep->SetReplayFlags(TTD::Replay::ReplayFlags::ReplaySegmentsSequentially |
|
||||
TTD::Replay::ReplayFlags::ReplayAllSegmentsWithoutFiltering);
|
||||
sweep->SetPosition(TTD::Replay::Position::Min);
|
||||
std::cerr << "[+] Beginning execution sweep...\n";
|
||||
sweep->ReplayForward();
|
||||
@@ -162,7 +248,8 @@ int wmain(int argc, wchar_t** argv) {
|
||||
if (limit_hit) {
|
||||
std::cerr << "[!] Reached --max-calls limit (" << opt.max_calls << ")\n";
|
||||
}
|
||||
std::cerr << "[+] Recorded " << g_report.process.calls.size() << " API calls\n";
|
||||
std::cerr << "[+] Recorded " << g_report.process.calls.size() << " API calls from "
|
||||
<< events_seen << " call/return events\n";
|
||||
|
||||
writeReport(opt.output);
|
||||
|
||||
|
||||
+117
-1
@@ -48,13 +48,30 @@ namespace ttdcapa {
|
||||
return resolvedTraceModuleExports;
|
||||
}
|
||||
|
||||
std::string convertWstringToString(const std::wstring& ws) {
|
||||
if (ws.empty()) {
|
||||
return {};
|
||||
}
|
||||
int needed = ::WideCharToMultiByte(CP_UTF8, 0, ws.data(), static_cast<int>(ws.size()), nullptr, 0, nullptr, nullptr);
|
||||
if (needed <= 0) {
|
||||
return {};
|
||||
}
|
||||
std::string out(static_cast<size_t>(needed), '\0');
|
||||
::WideCharToMultiByte(CP_UTF8, 0, ws.data(), static_cast<int>(ws.size()), out.data(), needed, nullptr, nullptr);
|
||||
return out;
|
||||
}
|
||||
|
||||
size_t readMemory(TTD::Replay::UniqueCursor* cursor, TTD::GuestAddress addr, void* dest, unsigned __int64 size) {
|
||||
TTD::Replay::MemoryBuffer memoryBuffer = cursor->get()->QueryMemoryBuffer(addr, TTD::BufferView{ dest, size });
|
||||
return memoryBuffer.Memory.Size;
|
||||
}
|
||||
|
||||
bool initializeTTDEngine(TTD::Replay::UniqueReplayEngine& engine, std::wstring traceFilePath) {
|
||||
ConsoleErrorReporting reporter;
|
||||
// The engine holds this pointer for its whole lifetime and calls into it
|
||||
// whenever it reports an error. A stack local would dangle the moment this
|
||||
// function returns, and the first derailment during ReplayForward would then
|
||||
// dispatch a virtual call through reclaimed stack memory.
|
||||
static ConsoleErrorReporting reporter;
|
||||
engine->RegisterDebugModeAndLogging(TTD::Replay::DebugModeType::None, &reporter);
|
||||
|
||||
if (!engine->Initialize(traceFilePath.c_str())) {
|
||||
@@ -132,6 +149,105 @@ namespace ttdcapa {
|
||||
return std::nullopt;
|
||||
}
|
||||
|
||||
namespace {
|
||||
// Shared by both typed readers: an address below the first 64 KiB is never a
|
||||
// valid user-mode string pointer, and treating one as such is how a flags
|
||||
// DWORD ends up rendered as text.
|
||||
constexpr uint64_t kMinStringAddr = 0x10000;
|
||||
|
||||
bool isPlausibleTextByte(unsigned char c) {
|
||||
return c == '\t' || c == '\r' || c == '\n' || (c >= 0x20 && c != 0x7f);
|
||||
}
|
||||
|
||||
// The -A entry points take strings in the ANSI code page, so their high
|
||||
// bytes are not UTF-8. Emitting them raw would produce a report that the
|
||||
// JSON serializer refuses to write, so transcode before anything else sees
|
||||
// them. Pure-ASCII input (the overwhelming majority) short-circuits.
|
||||
std::string ansiToUtf8(std::string bytes) {
|
||||
bool ascii = true;
|
||||
for (unsigned char c : bytes) {
|
||||
if (c >= 0x80) {
|
||||
ascii = false;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (ascii) {
|
||||
return bytes;
|
||||
}
|
||||
|
||||
int wide = ::MultiByteToWideChar(CP_ACP, 0, bytes.data(), static_cast<int>(bytes.size()), nullptr, 0);
|
||||
if (wide > 0) {
|
||||
std::wstring ws(static_cast<size_t>(wide), L'\0');
|
||||
if (::MultiByteToWideChar(CP_ACP, 0, bytes.data(), static_cast<int>(bytes.size()), ws.data(), wide) == wide) {
|
||||
return convertWstringToString(ws);
|
||||
}
|
||||
}
|
||||
|
||||
// Unconvertible: keep the ASCII skeleton rather than dropping the string.
|
||||
for (char& c : bytes) {
|
||||
if (static_cast<unsigned char>(c) >= 0x80) {
|
||||
c = '?';
|
||||
}
|
||||
}
|
||||
return bytes;
|
||||
}
|
||||
} // namespace
|
||||
|
||||
std::optional<std::string> readAnsiString(TTD::Replay::IThreadView const* thread, uint64_t addr, size_t maxChars) {
|
||||
if (addr < kMinStringAddr) {
|
||||
return std::nullopt;
|
||||
}
|
||||
std::vector<char> buf(maxChars + 1);
|
||||
auto result = thread->QueryMemoryBuffer(TTD::GuestAddress{ addr }, TTD::BufferView{ buf.data(), maxChars });
|
||||
size_t avail = result.Memory.Size;
|
||||
if (avail == 0) {
|
||||
return std::nullopt;
|
||||
}
|
||||
|
||||
std::string s;
|
||||
for (size_t i = 0; i < avail; ++i) {
|
||||
unsigned char c = static_cast<unsigned char>(buf[i]);
|
||||
if (c == 0) {
|
||||
return ansiToUtf8(std::move(s));
|
||||
}
|
||||
if (!isPlausibleTextByte(c)) {
|
||||
return std::nullopt; // control bytes mean this wasn't a string after all
|
||||
}
|
||||
s.push_back(static_cast<char>(c));
|
||||
}
|
||||
// Ran out of readable memory before the terminator; keep what we have as
|
||||
// long as it looked like text the whole way.
|
||||
return s.empty() ? std::nullopt : std::optional<std::string>(ansiToUtf8(std::move(s)));
|
||||
}
|
||||
|
||||
std::optional<std::string> readWideString(TTD::Replay::IThreadView const* thread, uint64_t addr, size_t maxChars) {
|
||||
if (addr < kMinStringAddr) {
|
||||
return std::nullopt;
|
||||
}
|
||||
std::vector<wchar_t> buf(maxChars + 1);
|
||||
auto result = thread->QueryMemoryBuffer(
|
||||
TTD::GuestAddress{ addr }, TTD::BufferView{ buf.data(), maxChars * sizeof(wchar_t) });
|
||||
size_t availChars = result.Memory.Size / sizeof(wchar_t);
|
||||
if (availChars == 0) {
|
||||
return std::nullopt;
|
||||
}
|
||||
|
||||
size_t len = 0;
|
||||
while (len < availChars && buf[len] != L'\0') {
|
||||
wchar_t wc = buf[len];
|
||||
// Reject C0 controls (other than the usual whitespace) rather than
|
||||
// emitting mojibake for a pointer that only looked like a string.
|
||||
if (wc < 0x20 && wc != L'\t' && wc != L'\r' && wc != L'\n') {
|
||||
return std::nullopt;
|
||||
}
|
||||
++len;
|
||||
}
|
||||
if (len == 0) {
|
||||
return std::string{};
|
||||
}
|
||||
return convertWstringToString(std::wstring(buf.data(), len));
|
||||
}
|
||||
|
||||
// Capture one candidate argument: a dereferenced string if it points to one, otherwise the raw integer value.
|
||||
ttdcapa::ArgValue captureCallArg(TTD::Replay::IThreadView const* thread, uint64_t value) {
|
||||
if (auto s = tryReadString(thread, value)) {
|
||||
|
||||
+49
-1
@@ -14,6 +14,8 @@
|
||||
#include <TTD/IReplayEngineStl.h>
|
||||
#include <TTD/ErrorReporting.h>
|
||||
|
||||
#include "win32meta.hpp"
|
||||
|
||||
class ConsoleErrorReporting : public TTD::ErrorReporting {
|
||||
public:
|
||||
void __fastcall VPrintError(char const* fmt, va_list args) override {
|
||||
@@ -43,13 +45,49 @@ namespace ttdcapa {
|
||||
// One call argument: either an integer or a dereferenced string
|
||||
using ArgValue = std::variant<int64_t, std::string>;
|
||||
|
||||
// One parameter decoded with the help of the Win32 metadata index. `name` and
|
||||
// `type` point into the index blob, which outlives every record, so they cost
|
||||
// nothing to copy. Enum values keep their index rather than their decoded flag
|
||||
// names so millions of in-memory calls stay cheap; names are resolved once at
|
||||
// report-write time.
|
||||
struct DecodedArg {
|
||||
const char* name = nullptr;
|
||||
const char* type = nullptr;
|
||||
win32meta::ArgKind kind = win32meta::ArgKind::Unknown;
|
||||
uint32_t enum_index = 0xFFFFFFFFu;
|
||||
uint64_t raw = 0; // the register/stack value as captured
|
||||
uint64_t deref = 0; // pointee, for PtrToInt and friends
|
||||
double fval = 0.0; // for Float/Double params (read from XMM)
|
||||
std::string str; // decoded string contents
|
||||
std::vector<uint8_t> bytes; // bounded buffer preview
|
||||
bool has_str = false;
|
||||
bool has_deref = false;
|
||||
bool has_fval = false;
|
||||
bool is_out = false;
|
||||
bool from_return = false; // contents were read at the return position
|
||||
};
|
||||
|
||||
// A dereference deferred until the call returns, so [Out] parameters can be
|
||||
// rendered filled in -- something only a time-travel trace makes easy.
|
||||
struct PendingOut {
|
||||
uint16_t param_index = 0;
|
||||
win32meta::ArgKind kind = win32meta::ArgKind::Unknown;
|
||||
uint64_t ptr = 0;
|
||||
uint16_t pointee_size = 0;
|
||||
win32meta::AuxKind aux_kind = win32meta::AuxKind::None;
|
||||
int32_t aux_value = 0;
|
||||
uint64_t in_cap = 0; // caller-supplied upper bound on length, 0 if unknown
|
||||
};
|
||||
|
||||
struct CallRecord {
|
||||
uint64_t tid = 0; // TTD Thread ID
|
||||
uint64_t seq = 0; // monotonic record order (consistent with timeline order)
|
||||
std::string position; // TTD navigable position "Sequence:Steps" (hex), for WinDbg
|
||||
std::wstring module; // resolved owning module, e.g. "kernel32" (no extension)
|
||||
std::string api; // resolved export name, e.g. "CreateFileA"
|
||||
std::vector<ArgValue> args;
|
||||
std::vector<ArgValue> args; // flat view consumed by capa (ints and strings)
|
||||
std::vector<DecodedArg> params; // rich view; only meaningful when `metadata` is set
|
||||
bool metadata = false; // args came from a real signature, not the heuristic
|
||||
bool has_ret = false;
|
||||
uint64_t ret = 0;
|
||||
};
|
||||
@@ -78,12 +116,22 @@ namespace ttdcapa {
|
||||
// TTD memory read utility function
|
||||
size_t readMemory(TTD::Replay::UniqueCursor* cursor, TTD::GuestAddress addr, void* dest, unsigned __int64 size);
|
||||
|
||||
// UTF-16 -> UTF-8, for the wide strings TTD and the PE headers hand back
|
||||
std::string convertWstringToString(const std::wstring& ws);
|
||||
|
||||
// Initializes the TTD engine based off a given trace file (.run) path
|
||||
bool initializeTTDEngine(TTD::Replay::UniqueReplayEngine& engine, std::wstring trace_file_path);
|
||||
|
||||
// Attempts to interpret the memory at a certain address as a string. If not a string, will return null
|
||||
std::optional<std::string> tryReadString(TTD::Replay::IThreadView const* thread, uint64_t addr);
|
||||
|
||||
// Read a NUL-terminated string the metadata told us is really there. Unlike
|
||||
// tryReadString these do not guess: no minimum length, and the wide reader
|
||||
// converts real UTF-16 (not just its ASCII subset) to UTF-8. Returns nullopt
|
||||
// only when the memory is unreadable or the bytes aren't a plausible string.
|
||||
std::optional<std::string> readAnsiString(TTD::Replay::IThreadView const* thread, uint64_t addr, size_t maxChars = 512);
|
||||
std::optional<std::string> readWideString(TTD::Replay::IThreadView const* thread, uint64_t addr, size_t maxChars = 512);
|
||||
|
||||
// Attempts to capture an argument as a string. If it doesn't look like a valid string, this function will return the same argument value
|
||||
ArgValue captureCallArg(TTD::Replay::IThreadView const* thread, uint64_t value);
|
||||
}
|
||||
|
||||
+62
-14
@@ -56,18 +56,6 @@ namespace ttdcapa {
|
||||
return result;
|
||||
}
|
||||
|
||||
std::string convertWstringToString(const std::wstring& ws) {
|
||||
if (ws.empty()) {
|
||||
return {};
|
||||
}
|
||||
int needed = ::WideCharToMultiByte(CP_UTF8, 0, ws.data(), static_cast<int>(ws.size()), nullptr, 0, nullptr, nullptr);
|
||||
if (needed <= 0) {
|
||||
return {};
|
||||
}
|
||||
std::string out(static_cast<size_t>(needed), '\0');
|
||||
::WideCharToMultiByte(CP_UTF8, 0, ws.data(), static_cast<int>(ws.size()), out.data(), needed, nullptr, nullptr);
|
||||
return out;
|
||||
}
|
||||
} // namespace
|
||||
|
||||
SampleHashes hashFile(const std::filesystem::path& path) {
|
||||
@@ -178,6 +166,18 @@ namespace ttdcapa {
|
||||
else if (a == L"--with-stack-args") {
|
||||
opt.with_stack_args = true;
|
||||
}
|
||||
else if (a == L"--win32-index" && i + 1 < argc) {
|
||||
opt.win32_index = argv[++i];
|
||||
}
|
||||
else if (a == L"--no-metadata") {
|
||||
opt.no_metadata = true;
|
||||
}
|
||||
else if (a == L"--max-buffer" && i + 1 < argc) {
|
||||
opt.max_buffer = static_cast<size_t>(std::wcstoull(argv[++i], nullptr, 10));
|
||||
}
|
||||
else if (a == L"--dump-sig" && i + 1 < argc) {
|
||||
opt.dump_sig = convertWstringToString(argv[++i]);
|
||||
}
|
||||
else if (!a.empty() && a[0] == L'-') {
|
||||
std::cerr << "unknown option\n";
|
||||
return false;
|
||||
@@ -190,7 +190,8 @@ namespace ttdcapa {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
return !opt.trace.empty();
|
||||
// --dump-sig is a metadata-only debug mode, so it doesn't need a trace.
|
||||
return !opt.trace.empty() || !opt.dump_sig.empty();
|
||||
}
|
||||
|
||||
bool writeReport(std::filesystem::path outputFilePath) {
|
||||
@@ -239,6 +240,9 @@ namespace ttdcapa {
|
||||
call["module"] = moduleStr;
|
||||
|
||||
call["api"] = callRecord.api;
|
||||
// Always an array, even when empty -- a zero-parameter function is a
|
||||
// real result, not a missing field.
|
||||
call["args"] = json::array();
|
||||
for (ArgValue& argValue : callRecord.args) {
|
||||
if (std::holds_alternative<int64_t>(argValue)) {
|
||||
call["args"].push_back(std::get<int64_t>(argValue));
|
||||
@@ -247,6 +251,46 @@ namespace ttdcapa {
|
||||
}
|
||||
}
|
||||
|
||||
// The rich, metadata-derived view. capa matches on "args"; this is for
|
||||
// ttd-timeline and human triage, so it can afford to be verbose. Absent
|
||||
// entirely when we had no signature, so consumers can tell "no
|
||||
// parameters" apart from "we didn't know".
|
||||
if (callRecord.metadata) {
|
||||
call["params"] = json::array();
|
||||
}
|
||||
for (DecodedArg& decoded : callRecord.params) {
|
||||
json param;
|
||||
param["name"] = decoded.name ? decoded.name : "";
|
||||
param["type"] = decoded.type ? decoded.type : "";
|
||||
param["kind"] = win32meta::kindName(decoded.kind);
|
||||
param["value"] = decoded.raw;
|
||||
if (decoded.has_str) {
|
||||
param["str"] = decoded.str;
|
||||
}
|
||||
if (decoded.has_deref) {
|
||||
param["deref"] = decoded.deref;
|
||||
}
|
||||
if (decoded.has_fval) {
|
||||
param["float"] = decoded.fval;
|
||||
}
|
||||
if (decoded.enum_index != 0xFFFFFFFFu) {
|
||||
std::vector<std::string> names = win32meta::index().decodeEnum(decoded.enum_index, decoded.raw);
|
||||
if (!names.empty()) {
|
||||
param["flags"] = names;
|
||||
}
|
||||
}
|
||||
if (!decoded.bytes.empty()) {
|
||||
param["bytes"] = to_hex(decoded.bytes);
|
||||
}
|
||||
if (decoded.is_out) {
|
||||
param["out"] = true;
|
||||
}
|
||||
if (decoded.from_return) {
|
||||
param["at_return"] = true;
|
||||
}
|
||||
call["params"].push_back(std::move(param));
|
||||
}
|
||||
|
||||
call["ret"] = callRecord.has_ret ? callRecord.ret : 0;
|
||||
|
||||
process["calls"].push_back(call);
|
||||
@@ -260,8 +304,12 @@ namespace ttdcapa {
|
||||
return false;
|
||||
}
|
||||
|
||||
reportFile << report.dump() << std::endl;
|
||||
// Recovered guest memory can always surprise us with a byte sequence that
|
||||
// isn't valid UTF-8. Substituting it beats throwing away an entire trace's
|
||||
// worth of extraction over one bad string.
|
||||
reportFile << report.dump(-1, ' ', false, nlohmann::json::error_handler_t::replace) << std::endl;
|
||||
reportFile.close();
|
||||
std::cerr << "DONE!\n";
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -33,7 +33,13 @@ namespace ttdcapa {
|
||||
std::filesystem::path trace;
|
||||
std::filesystem::path sample; // optional on-disk sample for hashing
|
||||
std::filesystem::path output; // empty will output to stdout
|
||||
std::filesystem::path win32_index; // empty means search the default locations
|
||||
std::string dump_sig; // print one signature and exit; no trace needed
|
||||
uint64_t max_calls = 0; // 0 means unlimited
|
||||
size_t max_buffer = 256; // bytes kept from any one counted buffer
|
||||
bool no_metadata = false; // force the pre-metadata heuristic capture
|
||||
// Only affects calls with no metadata: blindly grab four extra stack slots.
|
||||
// Functions we have a signature for always capture their true arity.
|
||||
bool with_stack_args = false;
|
||||
};
|
||||
|
||||
|
||||
@@ -0,0 +1,277 @@
|
||||
#include "win32meta.hpp"
|
||||
|
||||
#ifndef WIN32_LEAN_AND_MEAN
|
||||
#define WIN32_LEAN_AND_MEAN
|
||||
#endif
|
||||
#include <windows.h>
|
||||
|
||||
#include <algorithm>
|
||||
#include <cstring>
|
||||
#include <fstream>
|
||||
|
||||
namespace ttdcapa::win32meta {
|
||||
namespace {
|
||||
constexpr char kMagic[8] = { 'W', '3', '2', 'I', 'D', 'X', '0', '1' };
|
||||
constexpr uint32_t kFormatVersion = 1;
|
||||
|
||||
// Record sizes must match the struct.pack formats in tools/build-win32-index.py.
|
||||
constexpr size_t kHeaderSize = 32;
|
||||
constexpr size_t kFuncRecSize = 16;
|
||||
constexpr size_t kParamRecSize = 24;
|
||||
constexpr size_t kEnumRecSize = 16;
|
||||
constexpr size_t kEnumValRecSize = 16;
|
||||
|
||||
template <typename T>
|
||||
T readAt(const uint8_t* p, size_t off) {
|
||||
T v{};
|
||||
std::memcpy(&v, p + off, sizeof(T));
|
||||
return v;
|
||||
}
|
||||
|
||||
int popcount64(uint64_t v) {
|
||||
int n = 0;
|
||||
while (v) {
|
||||
v &= v - 1;
|
||||
++n;
|
||||
}
|
||||
return n;
|
||||
}
|
||||
} // namespace
|
||||
|
||||
bool Index::load(const std::filesystem::path& path, std::string& error) {
|
||||
std::ifstream f(path, std::ios::binary);
|
||||
if (!f) {
|
||||
error = "cannot open index";
|
||||
return false;
|
||||
}
|
||||
std::vector<uint8_t> blob((std::istreambuf_iterator<char>(f)), std::istreambuf_iterator<char>());
|
||||
if (blob.size() < kHeaderSize || std::memcmp(blob.data(), kMagic, sizeof(kMagic)) != 0) {
|
||||
error = "not a win32 index file (bad magic)";
|
||||
return false;
|
||||
}
|
||||
|
||||
const uint8_t* p = blob.data();
|
||||
uint32_t version = readAt<uint32_t>(p, 8);
|
||||
if (version != kFormatVersion) {
|
||||
error = "index format version " + std::to_string(version) +
|
||||
", expected " + std::to_string(kFormatVersion) + " (regenerate with tools/build-win32-index.py)";
|
||||
return false;
|
||||
}
|
||||
|
||||
uint32_t funcCount = readAt<uint32_t>(p, 12);
|
||||
uint32_t paramCount = readAt<uint32_t>(p, 16);
|
||||
uint32_t enumCount = readAt<uint32_t>(p, 20);
|
||||
uint32_t enumValCount = readAt<uint32_t>(p, 24);
|
||||
uint32_t strtabSize = readAt<uint32_t>(p, 28);
|
||||
|
||||
size_t funcOff = kHeaderSize;
|
||||
size_t paramOff = funcOff + static_cast<size_t>(funcCount) * kFuncRecSize;
|
||||
size_t enumOff = paramOff + static_cast<size_t>(paramCount) * kParamRecSize;
|
||||
size_t enumValOff = enumOff + static_cast<size_t>(enumCount) * kEnumRecSize;
|
||||
size_t strOff = enumValOff + static_cast<size_t>(enumValCount) * kEnumValRecSize;
|
||||
if (strOff + strtabSize != blob.size()) {
|
||||
error = "index is truncated or corrupt";
|
||||
return false;
|
||||
}
|
||||
// Every string offset is dereferenced without a bounds check below, so the
|
||||
// pool must be NUL-terminated for that to be safe.
|
||||
if (strtabSize == 0 || blob[blob.size() - 1] != 0) {
|
||||
error = "index string pool is not NUL-terminated";
|
||||
return false;
|
||||
}
|
||||
|
||||
blob_ = std::move(blob);
|
||||
p = blob_.data();
|
||||
const char* strtab = reinterpret_cast<const char*>(p + strOff);
|
||||
auto str = [&](uint32_t off) -> const char* {
|
||||
return off < strtabSize ? strtab + off : "";
|
||||
};
|
||||
|
||||
params_.resize(paramCount);
|
||||
for (uint32_t i = 0; i < paramCount; ++i) {
|
||||
size_t o = paramOff + static_cast<size_t>(i) * kParamRecSize;
|
||||
ParamSig& ps = params_[i];
|
||||
ps.name = str(readAt<uint32_t>(p, o));
|
||||
ps.type = str(readAt<uint32_t>(p, o + 4));
|
||||
ps.kind = static_cast<ArgKind>(p[o + 8]);
|
||||
ps.attrs = p[o + 9];
|
||||
ps.slot = p[o + 10];
|
||||
ps.auxKind = static_cast<AuxKind>(p[o + 11]);
|
||||
ps.auxValue = readAt<int32_t>(p, o + 12);
|
||||
ps.enumIndex = readAt<uint32_t>(p, o + 16);
|
||||
ps.pointeeSize = readAt<uint16_t>(p, o + 20);
|
||||
if (ps.enumIndex != 0xFFFFFFFFu && ps.enumIndex >= enumCount) {
|
||||
ps.enumIndex = 0xFFFFFFFFu;
|
||||
}
|
||||
}
|
||||
|
||||
funcs_.resize(funcCount);
|
||||
byName_.reserve(funcCount * 2);
|
||||
for (uint32_t i = 0; i < funcCount; ++i) {
|
||||
size_t o = funcOff + static_cast<size_t>(i) * kFuncRecSize;
|
||||
FuncSig& fs = funcs_[i];
|
||||
fs.name = str(readAt<uint32_t>(p, o));
|
||||
fs.dll = str(readAt<uint32_t>(p, o + 4));
|
||||
uint32_t firstParam = readAt<uint32_t>(p, o + 8);
|
||||
fs.paramCount = p[o + 12];
|
||||
fs.flags = p[o + 13];
|
||||
if (static_cast<size_t>(firstParam) + fs.paramCount > params_.size()) {
|
||||
error = "index parameter range out of bounds";
|
||||
return false;
|
||||
}
|
||||
fs.params = fs.paramCount ? ¶ms_[firstParam] : nullptr;
|
||||
byName_.emplace(std::string_view(fs.name), i);
|
||||
}
|
||||
|
||||
enumValues_.resize(enumValCount);
|
||||
for (uint32_t i = 0; i < enumValCount; ++i) {
|
||||
size_t o = enumValOff + static_cast<size_t>(i) * kEnumValRecSize;
|
||||
enumValues_[i].name = str(readAt<uint32_t>(p, o));
|
||||
enumValues_[i].value = readAt<int64_t>(p, o + 8);
|
||||
}
|
||||
|
||||
enums_.resize(enumCount);
|
||||
for (uint32_t i = 0; i < enumCount; ++i) {
|
||||
size_t o = enumOff + static_cast<size_t>(i) * kEnumRecSize;
|
||||
EnumTable& et = enums_[i];
|
||||
et.name = str(readAt<uint32_t>(p, o));
|
||||
et.valueOffset = readAt<uint32_t>(p, o + 4);
|
||||
et.valueCount = readAt<uint32_t>(p, o + 8);
|
||||
et.isFlags = p[o + 12] != 0;
|
||||
et.width = p[o + 13];
|
||||
if (static_cast<size_t>(et.valueOffset) + et.valueCount > enumValues_.size()) {
|
||||
et.valueOffset = 0;
|
||||
et.valueCount = 0;
|
||||
}
|
||||
}
|
||||
|
||||
path_ = path;
|
||||
return true;
|
||||
}
|
||||
|
||||
const FuncSig* Index::lookup(std::string_view api) const {
|
||||
auto it = byName_.find(api);
|
||||
return it == byName_.end() ? nullptr : &funcs_[it->second];
|
||||
}
|
||||
|
||||
const char* Index::enumName(uint32_t enumIndex) const {
|
||||
return enumIndex < enums_.size() ? enums_[enumIndex].name : "";
|
||||
}
|
||||
|
||||
std::vector<std::string> Index::decodeEnum(uint32_t enumIndex, uint64_t value) const {
|
||||
std::vector<std::string> out;
|
||||
if (enumIndex >= enums_.size()) {
|
||||
return out;
|
||||
}
|
||||
const EnumTable& et = enums_[enumIndex];
|
||||
// The captured value is a full 64-bit register; mask to the enum's real width
|
||||
// so sign-extension and upper garbage don't defeat the comparisons.
|
||||
uint64_t mask = et.width >= 8 ? ~0ull : ((1ull << (et.width * 8)) - 1);
|
||||
uint64_t v = value & mask;
|
||||
|
||||
const EnumValue* vals = enumValues_.data() + et.valueOffset;
|
||||
for (uint32_t i = 0; i < et.valueCount; ++i) {
|
||||
if ((static_cast<uint64_t>(vals[i].value) & mask) == v) {
|
||||
out.emplace_back(vals[i].name);
|
||||
return out; // exact match wins, flags or not
|
||||
}
|
||||
}
|
||||
if (!et.isFlags || v == 0) {
|
||||
return out;
|
||||
}
|
||||
|
||||
// Greedy decomposition: consume the widest matching bit groups first so
|
||||
// composites like GENERIC_WRITE beat their individual constituent bits.
|
||||
std::vector<uint32_t> order(et.valueCount);
|
||||
for (uint32_t i = 0; i < et.valueCount; ++i) {
|
||||
order[i] = i;
|
||||
}
|
||||
std::sort(order.begin(), order.end(), [&](uint32_t a, uint32_t b) {
|
||||
return popcount64(static_cast<uint64_t>(vals[a].value) & mask) >
|
||||
popcount64(static_cast<uint64_t>(vals[b].value) & mask);
|
||||
});
|
||||
|
||||
uint64_t remaining = v;
|
||||
for (uint32_t i : order) {
|
||||
uint64_t bits = static_cast<uint64_t>(vals[i].value) & mask;
|
||||
if (bits != 0 && (remaining & bits) == bits) {
|
||||
out.emplace_back(vals[i].name);
|
||||
remaining &= ~bits;
|
||||
}
|
||||
}
|
||||
if (remaining != 0) {
|
||||
char buf[32];
|
||||
std::snprintf(buf, sizeof(buf), "0x%llx", static_cast<unsigned long long>(remaining));
|
||||
out.emplace_back(buf);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
Index& index() {
|
||||
static Index instance;
|
||||
return instance;
|
||||
}
|
||||
|
||||
namespace {
|
||||
std::filesystem::path executableDir() {
|
||||
wchar_t buf[MAX_PATH * 4];
|
||||
DWORD n = ::GetModuleFileNameW(nullptr, buf, static_cast<DWORD>(std::size(buf)));
|
||||
if (n == 0 || n >= std::size(buf)) {
|
||||
return {};
|
||||
}
|
||||
return std::filesystem::path(buf, buf + n).parent_path();
|
||||
}
|
||||
} // namespace
|
||||
|
||||
bool loadIndex(const std::filesystem::path& explicitPath, std::string& error) {
|
||||
std::vector<std::filesystem::path> candidates;
|
||||
if (!explicitPath.empty()) {
|
||||
candidates.push_back(explicitPath);
|
||||
} else {
|
||||
std::filesystem::path dir = executableDir();
|
||||
if (!dir.empty()) {
|
||||
candidates.push_back(dir / L"win32-index.bin");
|
||||
// running straight out of ttd\bin\<plat>\<config>\ during development
|
||||
candidates.push_back(dir / L".." / L".." / L".." / L"data" / L"win32-index.bin");
|
||||
}
|
||||
}
|
||||
|
||||
std::error_code ec;
|
||||
for (const auto& c : candidates) {
|
||||
if (!std::filesystem::exists(c, ec)) {
|
||||
continue;
|
||||
}
|
||||
if (index().load(c, error)) {
|
||||
return true;
|
||||
}
|
||||
return false; // found but unusable: surface the real reason
|
||||
}
|
||||
error = "win32-index.bin not found (run tools/build-win32-index.py, or pass --win32-index)";
|
||||
return false;
|
||||
}
|
||||
|
||||
const char* kindName(ArgKind kind) {
|
||||
switch (kind) {
|
||||
case ArgKind::Integer: return "int";
|
||||
case ArgKind::Bool: return "bool";
|
||||
case ArgKind::Handle: return "handle";
|
||||
case ArgKind::Enum: return "enum";
|
||||
case ArgKind::Float: return "float";
|
||||
case ArgKind::Double: return "double";
|
||||
case ArgKind::AnsiString: return "str";
|
||||
case ArgKind::WideString: return "wstr";
|
||||
case ArgKind::AnsiBuffer: return "strbuf";
|
||||
case ArgKind::WideBuffer: return "wstrbuf";
|
||||
case ArgKind::ByteBuffer: return "buf";
|
||||
case ArgKind::PtrToInt: return "int*";
|
||||
case ArgKind::StructPtr: return "struct*";
|
||||
case ArgKind::FuncPtr: return "fnptr";
|
||||
case ArgKind::Guid: return "guid";
|
||||
case ArgKind::Pointer: return "ptr";
|
||||
case ArgKind::PtrToAnsiString: return "str*";
|
||||
case ArgKind::PtrToWideString: return "wstr*";
|
||||
case ArgKind::Unknown:
|
||||
default: return "unknown";
|
||||
}
|
||||
}
|
||||
} // namespace ttdcapa::win32meta
|
||||
@@ -0,0 +1,156 @@
|
||||
#ifndef WIN32META_HPP
|
||||
#define WIN32META_HPP
|
||||
|
||||
// Loader for the pre-baked Win32 API metadata index (win32-index.bin) produced by
|
||||
// tools/build-win32-index.py from the win32json submodule.
|
||||
//
|
||||
// The index answers one question per recorded call: "given this export name, what
|
||||
// are its parameters?" -- count, names, types, direction, and how to decode each
|
||||
// one. That turns the extractor's blind 4-register grab into an exact capture.
|
||||
//
|
||||
// Lookup is keyed on the bare function name. Module is carried for display only:
|
||||
// API sets mean the metadata's "CreateFileW -> KERNEL32.dll" shows up in a trace as
|
||||
// KERNELBASE.dll!CreateFileW, so matching on module would lose most calls.
|
||||
// See WIN32JSON-TTD-INTEGRATION-NOTES.md section 4.
|
||||
|
||||
#include <cstdint>
|
||||
#include <filesystem>
|
||||
#include <string>
|
||||
#include <string_view>
|
||||
#include <unordered_map>
|
||||
#include <vector>
|
||||
|
||||
namespace ttdcapa::win32meta {
|
||||
|
||||
// How to decode one parameter. Values must match the K_* constants in
|
||||
// tools/build-win32-index.py.
|
||||
enum class ArgKind : uint8_t {
|
||||
Unknown = 0,
|
||||
Integer, // plain scalar, never dereferenced
|
||||
Bool,
|
||||
Handle, // opaque HANDLE/HKEY/HWND; pointer-sized but not a pointer
|
||||
Enum, // scalar with a symbolic value table (see enumIndex)
|
||||
Float, // 4-byte float, arrives in XMM<slot>
|
||||
Double, // 8-byte float, arrives in XMM<slot>
|
||||
AnsiString, // char*, NUL-terminated
|
||||
WideString, // wchar_t*, NUL-terminated
|
||||
AnsiBuffer, // char[], length from aux
|
||||
WideBuffer, // wchar_t[], length from aux
|
||||
ByteBuffer, // void*/byte[], length from aux
|
||||
PtrToInt, // pointer to a pointeeSize-byte scalar
|
||||
StructPtr, // pointer to a struct/union (not expanded in v1)
|
||||
FuncPtr,
|
||||
Guid, // pointer to a 16-byte GUID
|
||||
Pointer, // opaque pointer, contents unknown
|
||||
PtrToAnsiString, // char** -- out-param that receives an allocated string
|
||||
PtrToWideString, // wchar_t**
|
||||
};
|
||||
|
||||
// Bitmask of a parameter's direction/nullability attributes.
|
||||
enum ParamAttr : uint8_t {
|
||||
AttrIn = 0x01,
|
||||
AttrOut = 0x02,
|
||||
AttrOptional = 0x04,
|
||||
AttrConst = 0x08,
|
||||
AttrReserved = 0x10,
|
||||
AttrNotNulTerminated = 0x20,
|
||||
AttrNulNulTerminated = 0x40,
|
||||
AttrComOutPtr = 0x80,
|
||||
};
|
||||
|
||||
// Where a buffer parameter's length comes from.
|
||||
enum class AuxKind : uint8_t {
|
||||
None = 0,
|
||||
BytesFromParam, // auxValue is the index of a parameter holding a byte count
|
||||
CountFromParam, // auxValue is the index of a parameter holding an element count
|
||||
CountConst, // auxValue is the element count itself
|
||||
};
|
||||
|
||||
enum FuncFlag : uint8_t {
|
||||
FlagHiddenRetPtr = 0x01, // returns a large aggregate; RCX is the hidden return buffer
|
||||
FlagUnsupported = 0x02, // at least one parameter could not be classified
|
||||
FlagSetLastError = 0x04,
|
||||
};
|
||||
|
||||
struct ParamSig {
|
||||
const char* name = "";
|
||||
const char* type = "";
|
||||
ArgKind kind = ArgKind::Unknown;
|
||||
uint8_t attrs = 0;
|
||||
uint8_t slot = 0; // positional ABI slot, already shifted for FlagHiddenRetPtr
|
||||
AuxKind auxKind = AuxKind::None;
|
||||
int32_t auxValue = 0;
|
||||
uint32_t enumIndex = 0xFFFFFFFFu;
|
||||
uint16_t pointeeSize = 0; // bytes per pointee/element, 0 if unknown
|
||||
|
||||
bool isIn() const { return (attrs & AttrIn) != 0; }
|
||||
bool isOut() const { return (attrs & AttrOut) != 0; }
|
||||
bool isFloat() const { return kind == ArgKind::Float || kind == ArgKind::Double; }
|
||||
bool hasEnum() const { return enumIndex != 0xFFFFFFFFu; }
|
||||
};
|
||||
|
||||
struct FuncSig {
|
||||
const char* name = "";
|
||||
const char* dll = "";
|
||||
const ParamSig* params = nullptr;
|
||||
uint8_t paramCount = 0;
|
||||
uint8_t flags = 0;
|
||||
|
||||
bool unsupported() const { return (flags & FlagUnsupported) != 0; }
|
||||
bool hiddenRetPtr() const { return (flags & FlagHiddenRetPtr) != 0; }
|
||||
};
|
||||
|
||||
// Loaded once at startup and read-only thereafter, so the replay sweep can hit
|
||||
// it from the call callback without synchronisation.
|
||||
class Index {
|
||||
public:
|
||||
bool load(const std::filesystem::path& path, std::string& error);
|
||||
bool loaded() const { return !funcs_.empty(); }
|
||||
size_t functionCount() const { return funcs_.size(); }
|
||||
const std::filesystem::path& path() const { return path_; }
|
||||
|
||||
// Bare export name, e.g. "CreateFileW". Returns nullptr when unknown, which
|
||||
// is the caller's cue to fall back to the heuristic capture.
|
||||
const FuncSig* lookup(std::string_view api) const;
|
||||
|
||||
const char* enumName(uint32_t enumIndex) const;
|
||||
|
||||
// Symbolic names for `value`: an exact match if one exists, otherwise a
|
||||
// greedy bit decomposition for flag enums. Empty when nothing matches.
|
||||
std::vector<std::string> decodeEnum(uint32_t enumIndex, uint64_t value) const;
|
||||
|
||||
private:
|
||||
struct EnumTable {
|
||||
const char* name = "";
|
||||
uint32_t valueOffset = 0;
|
||||
uint32_t valueCount = 0;
|
||||
bool isFlags = false;
|
||||
uint8_t width = 4;
|
||||
};
|
||||
struct EnumValue {
|
||||
const char* name = "";
|
||||
int64_t value = 0;
|
||||
};
|
||||
|
||||
std::vector<uint8_t> blob_;
|
||||
std::vector<FuncSig> funcs_;
|
||||
std::vector<ParamSig> params_;
|
||||
std::vector<EnumTable> enums_;
|
||||
std::vector<EnumValue> enumValues_;
|
||||
std::unordered_map<std::string_view, uint32_t> byName_;
|
||||
std::filesystem::path path_;
|
||||
};
|
||||
|
||||
// Process-wide instance; empty until loadIndex() succeeds.
|
||||
Index& index();
|
||||
|
||||
// Try `explicitPath` if non-empty, else the conventional locations next to the
|
||||
// executable and in the source tree. Returns false with `error` set; the caller
|
||||
// is expected to warn and continue in heuristic mode.
|
||||
bool loadIndex(const std::filesystem::path& explicitPath, std::string& error);
|
||||
|
||||
const char* kindName(ArgKind kind);
|
||||
|
||||
} // namespace ttdcapa::win32meta
|
||||
|
||||
#endif
|
||||
@@ -70,15 +70,19 @@
|
||||
</Link>
|
||||
</ItemDefinitionGroup>
|
||||
<ItemGroup>
|
||||
<ClCompile Include="src\abi_x64.cpp" />
|
||||
<ClCompile Include="src\main.cpp" />
|
||||
<ClCompile Include="src\ttd_pe_utils.cpp" />
|
||||
<ClCompile Include="src\ttdutils.cpp" />
|
||||
<ClCompile Include="src\utils.cpp" />
|
||||
<ClCompile Include="src\win32meta.cpp" />
|
||||
</ItemGroup>
|
||||
<ItemGroup>
|
||||
<ClInclude Include="src\abi_x64.hpp" />
|
||||
<ClInclude Include="src\ttd_pe_utils.hpp" />
|
||||
<ClInclude Include="src\ttdutils.hpp" />
|
||||
<ClInclude Include="src\utils.hpp" />
|
||||
<ClInclude Include="src\win32meta.hpp" />
|
||||
</ItemGroup>
|
||||
<ItemGroup>
|
||||
<None Include="packages.config" />
|
||||
@@ -101,6 +105,13 @@
|
||||
</ItemGroup>
|
||||
<Copy SourceFiles="@(TTDRuntime)" DestinationFolder="$(OutDir)" SkipUnchangedFiles="true" Condition="'@(TTDRuntime)'!=''" ContinueOnError="true" />
|
||||
</Target>
|
||||
<!-- The Win32 metadata index the argument decoder reads at startup. Regenerate
|
||||
with tools\build-win32-index.py after bumping the win32json submodule; the
|
||||
extractor still runs (heuristically) if it is absent. -->
|
||||
<Target Name="CopyWin32Index" AfterTargets="Build">
|
||||
<Copy SourceFiles="data\win32-index.bin" DestinationFolder="$(OutDir)" SkipUnchangedFiles="true" Condition="Exists('data\win32-index.bin')" />
|
||||
<Warning Condition="!Exists('data\win32-index.bin')" Text="data\win32-index.bin is missing; run 'python tools\build-win32-index.py' to enable metadata-driven argument decoding." />
|
||||
</Target>
|
||||
<Import Project="packages\nlohmann.json.3.12.0\build\native\nlohmann.json.targets" Condition="Exists('packages\nlohmann.json.3.12.0\build\native\nlohmann.json.targets')" />
|
||||
<Target Name="EnsureNuGetPackageBuildImports" BeforeTargets="PrepareForBuild">
|
||||
<PropertyGroup>
|
||||
|
||||
@@ -1,9 +1,15 @@
|
||||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
|
||||
<ItemGroup>
|
||||
<ClCompile Include="src\abi_x64.cpp">
|
||||
<Filter>Source Files</Filter>
|
||||
</ClCompile>
|
||||
<ClCompile Include="src\main.cpp">
|
||||
<Filter>Source Files</Filter>
|
||||
</ClCompile>
|
||||
<ClCompile Include="src\win32meta.cpp">
|
||||
<Filter>Source Files</Filter>
|
||||
</ClCompile>
|
||||
<ClCompile Include="src\ttd_pe_utils.cpp">
|
||||
<Filter>Source Files</Filter>
|
||||
</ClCompile>
|
||||
@@ -15,9 +21,15 @@
|
||||
</ClCompile>
|
||||
</ItemGroup>
|
||||
<ItemGroup>
|
||||
<ClInclude Include="src\abi_x64.hpp">
|
||||
<Filter>Header Files</Filter>
|
||||
</ClInclude>
|
||||
<ClInclude Include="src\ttd_pe_utils.hpp">
|
||||
<Filter>Header Files</Filter>
|
||||
</ClInclude>
|
||||
<ClInclude Include="src\win32meta.hpp">
|
||||
<Filter>Header Files</Filter>
|
||||
</ClInclude>
|
||||
<ClInclude Include="src\ttdutils.hpp">
|
||||
<Filter>Header Files</Filter>
|
||||
</ClInclude>
|
||||
|
||||
Submodule
+1
Submodule win32json added at 071df49952
Reference in New Issue
Block a user