Document the .ttdb format

README.md

@@ -141,8 +141,9 @@ Useful flags:
 
 ## Binary reports
 `-o <path>` writes the JSON report capa consumes. `-b/--binary <path>` writes the same data in a
-compact, memory-mappable layout instead (see `ttd/src/binreport.hpp`), intended for tools that load
-a whole report and browse it. Either or both can be given.
+compact, memory-mappable layout instead, intended for tools that load a whole report and browse it.
+Either or both can be given. The layout is specified in [docs/ttdb-format.md](docs/ttdb-format.md);
+the writer is `ttd/src/binreport.cpp`.
 
 JSON is a poor fit for that second job. On a 3.4M-call trace, measured:
 
@@ -205,6 +206,25 @@ python ttd-timeline.py <JSON REPORT PATH> --calls
 ```
 
 # Limitations
+- **String and buffer arguments are missing from some calls, and this is expected.** The sweep
+  reads guest memory through the `IThreadView` the replay engine hands to a callback, and the SDK
+  fixes those reads to `QueryMemoryPolicy::ThreadLocal` -- "a quick query that concentrates on the
+  current position and current thread, possibly ignoring some of the memory observed by other
+  threads, along with memory observed by the current thread in the past or future". It is allowed
+  to return nothing even when the data is in the trace, and it does: about a quarter of string
+  parameters come back empty, and `LoadLibraryExW`'s path was missing on 39% of its calls in one
+  trace. Reading the same address at the same position through a *cursor* returns the whole
+  string, so this is a property of which interface the read goes through, not of the trace.
+
+  A missing string is therefore never evidence that the argument was null -- the parameter's raw
+  pointer value is still recorded, and is usually valid. The policy cannot be changed for a
+  thread view; `SetDefaultMemoryPolicy` on the cursor has no effect on it (measured:
+  byte-identical output).
+
+  Recovering them means going back over the failed addresses with a cursor afterwards, which
+  works -- about 77% come back -- but costs roughly 4x the total extraction time on a large
+  trace, because every seek replays from a keyframe. That is a large enough trade to want its
+  own discussion, so it is not in the extractor today.
 - x64 and x86 traces are supported, including WoW64; ARM64 is not. Bitness is decided per call
   from the PE header of the module owning the call target rather than once per trace -- a WoW64
   process runs both widths at once, and `SystemInfo.ProcessorArchitecture` describes the machine

docs/ttdb-format.md

@@ -0,0 +1,206 @@
+# The `.ttdb` binary report format
+
+A `.ttdb` file holds the Windows API calls extracted from a Time Travel Debugging trace:
+for each call, when it happened, which function was called, what its arguments were, and
+what it returned.
+
+It is written by `ttdcapa-extract -b out.ttdb` and is designed to be **memory-mapped and
+read in place**. Nothing is parsed on open and nothing is allocated per call, so opening a
+3.4-million-call report costs a header validation rather than the ~17 seconds the
+equivalent JSON took. The reference implementations are `ttd/src/binreport.cpp` (writer)
+and, in the Binary Ninja debugger, `core/ttdbehavior.cpp` (reader).
+
+Format version 2. All integers are **little-endian**. All *file offsets* are byte offsets
+from the start of the file, so a mapped view needs no fixups; *region offsets* (into the
+string table or the blob) are relative to that region's start, and are noted as such.
+
+## Layout
+
+```
++----------------------------+ 0
+| header                     | 128 bytes
++----------------------------+ callsOff
+| call records               | callCount * 48 bytes, in sweep order
++----------------------------+ paramsOff
+| parameter records          | variable length, referenced by call records
++----------------------------+ stringsOff
+| string table               | stringsSize bytes, NUL-terminated UTF-8
++----------------------------+ blobOff
+| blob                       | blobSize bytes: search text, strings, buffers
++----------------------------+ end of file
+```
+
+The split exists because the three kinds of data have different access patterns. Call
+records are fixed size so row *N* is a multiply. The string table is deduplicated, which is
+where the compression really comes from -- a trace has millions of calls but only thousands
+of distinct module and function names, so every name in a 3.4M-call report fits in about
+100 KB. Everything variable-length and unique to one call goes in the blob.
+
+## Header (128 bytes)
+
+| Offset | Type | Field | Notes |
+| ---: | --- | --- | --- |
+| 0 | `char[8]` | magic | `TTDBEHV1` |
+| 8 | `u32` | version | 2 |
+| 12 | `u32` | arch | 0 = x64, 1 = x86 |
+| 16 | `u64` | callCount | number of call records |
+| 24 | `u64` | paramBytes | size in bytes of the parameter region |
+| 32 | `u64` | pid | process id of the traced process |
+| 40 | `u64` | callsOff | file offset of the call records |
+| 48 | `u64` | paramsOff | file offset of the parameter records |
+| 56 | `u64` | stringsOff | file offset of the string table |
+| 64 | `u64` | stringsSize | size in bytes of the string table |
+| 72 | `u64` | blobOff | file offset of the blob |
+| 80 | `u64` | blobSize | size in bytes of the blob |
+| 88 | `u64` | decodedCount | calls whose parameters came from a real signature |
+| 96 | `u64` | maxSeq | highest sequence number, i.e. `callCount - 1` |
+| 104 | `u32` | tracePathStr | string-table offset of the source `.run` path |
+| 108 | `u32` | sampleNameStr | string-table offset of the main module's name |
+| 112 | `u32` | maxPositionChars | widest rendered position, for column sizing |
+| 116 | | *(reserved)* | zero |
+
+`arch` describes the **traced process**, taken from the main module's PE format, not the
+machine that recorded the trace. A reader should reject a file whose magic does not match,
+whose version it does not know, or whose regions do not fit inside the file.
+
+## Call record (48 bytes)
+
+| Offset | Type | Field | Notes |
+| ---: | --- | --- | --- |
+| 0 | `u64` | ret | the call's return value |
+| 8 | `u32` | tid | TTD unique thread id |
+| 12 | `u32` | positionSequence | TTD position, sequence part |
+| 16 | `u32` | positionSteps | TTD position, steps part |
+| 20 | `u32` | moduleStr | string-table offset of the module name, e.g. `kernel32` |
+| 24 | `u32` | apiStr | string-table offset of the function name, e.g. `CreateFileW` |
+| 28 | `u32` | paramOff | offset into the parameter region; 0 when there are none |
+| 32 | `u32` | searchOff | offset into the blob of this call's search text |
+| 36 | `u16` | paramCount | low 15 bits; bit 15 (`0x8000`) set means *decoded* |
+| 38 | `u16` | searchLen | length in bytes of the search text |
+| 40 | `u64` | returnAddress | the instruction after the CALL, i.e. the call site |
+
+**There is no sequence number field.** Recorded calls are numbered densely from zero in
+sweep order, so a record's index *is* its sequence number.
+
+**Position** is conventionally rendered `%X:%X` of sequence and steps -- `45905:15A5` --
+which is the form WinDbg and the Binary Ninja debugger accept for time travel.
+
+**`returnAddress`** identifies the caller, which is how you distinguish a call the sample
+made itself from one a system DLL made on its behalf.
+
+**The decoded bit** matters for interpreting the parameters. When set, the extractor had a
+real signature for the function from Microsoft's Win32 metadata, so the parameters have
+correct arity, names, and types. When clear, it fell back to capturing argument registers
+(x64) or stack slots (x86) heuristically: the values are positional, unnamed, and the count
+is a guess. A decoded call may legitimately have zero parameters, which is why the flag is
+separate from the count.
+
+## Parameter record (variable length)
+
+Parameters for one call are stored consecutively starting at `paramsOff + paramOff`, and
+are decoded in sequence -- there is no index, so reading parameter *k* means walking the
+*k* before it. That is deliberate: a viewer only decodes parameters for rows a user
+actually looks at.
+
+Fixed part, 18 bytes:
+
+| Offset | Type | Field |
+| ---: | --- | --- |
+| 0 | `u8` | kind |
+| 1 | `u8` | bits |
+| 2 | `u32` | nameStr (string-table offset; 0 when unnamed) |
+| 6 | `u32` | typeStr (string-table offset; 0 when untyped) |
+| 10 | `u64` | value (the raw register or stack value) |
+
+Then, in this order, only the parts whose bit is set in `bits`:
+
+| Bit | Name | Adds |
+| ---: | --- | --- |
+| 0x01 | Out | *(nothing; marks an `[Out]` parameter)* |
+| 0x02 | AtReturn | *(nothing; value was re-read at the call's return)* |
+| 0x04 | HasDeref | `u64` deref -- the pointed-to value |
+| 0x08 | HasStr | `u32` strOff, `u32` strLen -- blob offset and length |
+| 0x10 | HasBytes | `u32` bytesOff, `u32` bytesLen, `u64` bytesTotal |
+| 0x20 | HasFlags | `u32` flagsOff, `u32` flagsLen -- `|`-joined names in the blob |
+
+So a parameter's size is 18 bytes plus 8 for HasDeref, 8 for HasStr, 16 for HasBytes and 8
+for HasFlags, in that order.
+
+**`bytesTotal`** is the buffer's real length when only a prefix was captured (the capture
+limit defaults to 64 KiB); `bytesLen` is what is actually in the file. `bytesTotal >
+bytesLen` means truncated.
+
+**`AtReturn`** marks values re-read at the call's return position rather than at the call.
+This is what makes `[Out]` parameters renderable at all -- at the moment of the call they
+have not been written yet -- and it is the one thing a time-travel trace gives you that a
+live debugger cannot easily.
+
+### Parameter kinds
+
+`kind` is an index into this list. It describes how the value should be interpreted, not
+its C type, which is in `typeStr`.
+
+| # | Name | Meaning |
+| ---: | --- | --- |
+| 0 | *(unknown)* | unclassified, or a heuristic capture with no signature |
+| 1 | `int` | plain scalar |
+| 2 | `bool` | |
+| 3 | `handle` | opaque, pointer-sized, never dereferenced |
+| 4 | `enum` | scalar with a symbolic value table; see HasFlags |
+| 5 | `float` | |
+| 6 | `double` | |
+| 7 | `str` | `char*`, NUL-terminated |
+| 8 | `wstr` | `wchar_t*`, NUL-terminated |
+| 9 | `strbuf` | `char[]`, length from a sibling parameter |
+| 10 | `wstrbuf` | `wchar_t[]` |
+| 11 | `buf` | `void*`/byte array |
+| 12 | `int*` | pointer to a scalar; see HasDeref |
+| 13 | `struct*` | pointer to a struct or union, not expanded |
+| 14 | `fnptr` | |
+| 15 | `guid` | pointer to a 16-byte GUID, rendered into HasStr |
+| 16 | `ptr` | opaque pointer |
+| 17 | `str*` | `char**`, an out-parameter receiving an allocated string |
+| 18 | `wstr*` | `wchar_t**` |
+
+## String table
+
+A run of NUL-terminated UTF-8 strings, deduplicated. A string-table offset is relative to
+`stringsOff`. **Offset 0 is the empty string**, so 0 doubles as "absent".
+
+Only repeated text lives here: module names, function names, parameter names, parameter
+type names.
+
+## Blob
+
+Raw bytes, referenced by `(offset, length)` pairs relative to `blobOff`. It holds three
+things, interleaved in whatever order the writer emitted them:
+
+- **Search text**, one per call, referenced by `searchOff`/`searchLen`. This is a
+  lowercased rendering of `module!api` followed by each parameter as it would be displayed,
+  including printable runs extracted from captured buffers. It exists so a text filter is a
+  substring scan over mapped pages with nothing to build first.
+- **Parameter strings**, referenced by HasStr.
+- **Captured buffer contents**, referenced by HasBytes. Raw bytes, not hex.
+
+## Reading a report
+
+Validate the magic and version, check the regions fit in the file, and map it. Then:
+
+- **Row *N***: read 48 bytes at `callsOff + N * 48`.
+- **Its module and function**: NUL-terminated strings at `stringsOff + moduleStr` and
+  `stringsOff + apiStr`.
+- **Its parameters**: walk `paramCount & 0x7FFF` records from `paramsOff + paramOff`,
+  decoding each per the bits.
+- **Filtering**: for a text match, `memmem` the needle in the `searchLen` bytes at
+  `blobOff + searchOff`. For a match on module or function, resolve the name against the
+  string table once to a set of offsets, then compare `moduleStr`/`apiStr` as integers --
+  that is far cheaper than a string search, which is why scoped queries are faster than
+  plain text ones.
+
+## A caveat about missing data
+
+A string parameter whose HasStr bit is clear does **not** mean the argument was null. The
+extractor's sweep reads guest memory through an interface the TTD SDK restricts to a fast,
+incomplete lookup, which may return nothing even when the data is present in the trace;
+roughly a quarter of string parameters are affected. The parameter's raw `value` is still
+recorded and is usually a valid pointer. See the Limitations section of the README.