One document now covers both what ttdcapa-extract does and what it writes,
since anyone reading either needs the other. Renamed to match the tool it
documents, which also reads better in the release artifact next to the
executable.
Brings the README's prerequisites and DLL instructions in line with the
toolset and --ttd-dlls changes.
TTDReplay.dll and TTDReplayCPU.dll are Microsoft's, shipped with WinDbg and
not ours to redistribute, so requiring them beside the executable made the
extractor awkward to ship. --ttd-dlls points at wherever WinDbg already put
them. TTDReplay.dll is delay-loaded so this can take effect at runtime;
without the flag the normal search order applies, which finds copies beside
the executable exactly as before. Either way the load is explicit, so a
missing engine is a clear message rather than the structured exception a
failed delay-load raises.
The toolset is no longer pinned to v145, which only VS2026 has -- it
defaults to v143 and can be overridden.
CI builds Release x64 on windows-2022, smoke-tests --dump-sig, and uploads
ttdcapa-extract.exe, win32-index.bin and docs/ttdb-format.md. Tags publish
the same set as a release. The replay DLLs are deliberately absent from it.
A buffer parameter contributed only its pointer to the searchable text --
'lpbuffer=0x4fdf54' -- so searching for a string a program actually wrote
or read found nothing, even though the bytes were sitting in the report.
Append the printable runs from each captured buffer instead, bounded at 96
characters so a 64 KiB buffer cannot inflate every haystack.
The WriteFile that prints hello now reads
kernelbase!writefile hfile=0xc lpbuffer=0x4fdf54 hello, world! ...
and matches a search for it. No measurable size cost: buffer parameters are
well under 1% of calls.
The replay engine already hands the fall-through address to the call
callback -- it is what we match the return against -- so recording it is a
single store and no extra memory reads. Measured on the 3.4M-call netac
trace: 34.5s and 32.0s against a 35.0s baseline, i.e. inside run-to-run
noise, with the report write unchanged at ~3.3s. The binary report grows
8 bytes per call, 585MB to 611MB.
Worth having because it names the call site: user32!MsgWaitForMultipleObjectsEx
in that trace returns to 0x00f91edc, in the sample's own range, where the
ntdll!_allmul calls return into ntdll itself. That is the difference between
a call the sample made and one made on its behalf.
Binary format version goes to 2; a reader checking the version will ask for
a re-extract rather than misread the 40-byte records.
Measured: the binary write is 3.6s against JSON's ~56s on a 3.4M-call
trace. Emitted both readably and as a [timing] line so the sidebar can put
it in front of the user, since the cost of the format was the thing worth
knowing.
JSON is the right output for capa and the wrong one for a tool that loads a
whole report and browses it. Measured on a 3.4M-call trace: of an 86s run,
the ~30s sweep was dwarfed by ~56s of JSON serialisation, and loading the
654MB result back cost 16.6s -- 8.9s of DOM parse plus 7.6s allocating
per-call objects -- of which only 138ms was disk I/O. Two thirds of the
wall clock was the format, and none of it was inherent.
-b/--binary writes the same data in a layout designed to be mapped and read
in place: fixed-size call records indexable by row, positions as two
integers rather than "45905:15A5", raw bytes rather than hex, a
deduplicated string table, and a prebuilt lowercased search haystack per
call so a filter needs nothing built first. -o still writes JSON; either or
both can be given.
Result: the same extraction takes 44s instead of 86s, and the file is
585MB. Loading is a memory map.
The string table is the standout -- every module and API name across 3.4M
calls fits in 100KB. The remaining bulk is parameter records (218MB) and
haystacks (237MB), both of which have obvious trims left if size starts to
matter more than time.
Two new flags, both for a UI driving this as a child process; interactive
use is unchanged.
--progress emits "[progress] <percent> <calls>" on stderr a few times a
second, plus "[phase] sweep" / "[phase] write" markers. The percentage
comes from the callback thread's position against the trace's last
sequence. Positions from different threads are not perfectly ordered, so
roughly 4% of samples read behind the previous one; they are clamped
forward rather than allowed to walk a progress bar backwards. The phases
matter because the sweep is only part of the wall clock -- on a 3.4M-call
trace it takes ~30s while serialising the 655MB report takes comparable
time, so a caller tracking only the sweep would sit at 100% looking wedged.
--cancel-on-stdin starts a watcher thread that calls InterruptReplay when
it reads "cancel", which the SDK documents as safe from any thread. The
report is then written as usual: everything recorded so far is already in
g_report, so a cancelled run yields a complete, valid report of a shorter
prefix of the trace. Verified stopping a netac sweep at 1.54M of 3.38M
calls, 317k of them with decoded parameters.
Measured on a 654 MB report of a 3.4M-call trace: 23k buffer parameters
totalling 1.1 MB, so buffers are nowhere near the dominant cost and the
old 256-byte cap was buying almost nothing. It was, however, cutting off
2014 buffers across 19 APIs -- ReadFile, WriteFile, RegGetValueW,
GetFontData and DeviceIoControl among them -- right where the interesting
content starts.
64 KiB covers typical socket and file reads whole.
A 4096-byte WriteFile showed up in the report as a plain 256-byte buffer,
with nothing to say the other 3840 bytes had been dropped. Emit
bytes_total alongside bytes whenever the cap actually bit, so a consumer
can distinguish a short buffer from the head of a long one.
Only when it really was cut short: a character buffer trimmed at its NUL
is complete, since the count parameter is the caller's capacity rather
than the string's length. Claiming truncation there would send someone
hunting for data that never existed.
Validated against a real 32-bit trace: NtOpenFile's FileHandle out-param
came back as 0x820077a1cf10, an 8-byte read of a 4-byte HANDLE that
spliced the following dword into the value. The index measures HANDLE at
its x64 width, so on a 32-bit guest an 8-byte pointee is almost always a
pointer-sized type and should be read as 4.
A genuine 64-bit pointee now loses its high half instead, which is still
better than a value mixed with unrelated memory. Distinguishing the two
needs the same index change the README already describes.
x64 output remains byte-identical on the reference trace.
Bitness is decided per call from the PE header of the module owning the
call target, not once per trace. SystemInfo.ProcessorArchitecture comes
from GetSystemInfo() on the recording machine, so it reports AMD64 for a
32-bit process recorded on an x64 box -- it never described the guest.
A WoW64 process runs both widths at once, so per-module is also the only
answer that can be right.
- ttd_pe_utils parses PE32 alongside PE32+, and reports each image's
bitness. Previously readNTHeaders rejected anything that was not
PE32+, so on a 32-bit trace the export map came back empty and no
calls were recorded at all.
- abi_x64 becomes abi, parameterised over the calling convention. x86
passes everything on the stack, so each parameter's offset depends on
the sizes of the ones before it; computeStackOffsets recovers those.
__stdcall and __cdecl are indistinguishable from the callee's entry,
so one layout serves both.
- Pointer-width dereferences (T** hops, untyped pointees) follow the
guest's width rather than a hardcoded 8.
- decodeArgs now declines signatures it cannot lay out, and the caller
falls back to the heuristic. That covers pointer-sized scalars, whose
x86 footprint the index cannot express, and by-value aggregates. See
the README for what this costs and how to fix it properly.
- The x86 heuristic fallback reads the stack, since nothing arrives in
registers there.
- --dump-sig shows the computed x86 offsets, so a layout can be checked
without a trace.
Also drops a duplicated thread-list collection that made every trace
report its threads twice.
x64 output is unchanged: byte-identical call records on a 4741-call
reference trace. The x86 path has not yet been run against a real
32-bit trace.
Win32 metadata index (win32json) drives per-call signature lookup so
arguments are captured at their true arity and type, [Out] params are
re-read at the call's return position, and buffers are dumped as bytes.
Jonathan Beierle