Defender Scan Pipeline — Stage 9

BRUTE Matching

Format-agnostic hash-string matching for polymorphic malware detection

0Signature Types
8Caller Contexts
10+HSTR Variants
3Feature Prefixes

SIGNATURE_TYPE_BRUTE @ 0x10986D8C • HSTR_WEIGHT @ 0x1097C6D0 • mpengine.dll

BRUTE vs. PEHSTR

Two complementary matching approaches: format-specific vs. format-agnostic

PEHSTR (Stage 3)

Format-aware: PE sections only
Targets .text, .data, imports, exports
SIGNATURE_TYPE_PEHSTR @ 0x109869C8
Fast: searches structured regions
Cannot see packed/encrypted content
Primary for known PE malware

BRUTE (Stage 9)

Format-agnostic: raw bytes
Targets the entire data stream
SIGNATURE_TYPE_BRUTE @ 0x10986D8C
Thorough: searches everything
Works on unpacked/deobfuscated content
Primary for polymorphic/script malware
PEHSTR catches fast at Stage 3 -- BRUTE catches deep at Stage 9 BRUTE sees content after emulation, extraction, and deobfuscation

HSTR Matching Engine

Rolling hash with weighted scoring -- the core of BRUTE matching

1. Slide rolling hash window over input bytes
2. Hash table lookup (O(1) per position)
3. Full string comparison to confirm match
4. Record (sig_id, offset, length, weight)
5. Accumulate weight per signature group
6. Threshold check: weight >= limit = MATCH
// Core matching loop for offset in 0..data.len() { let hash = rolling_hash( data, offset); let entry = hstr_lookup( &table, hash); if entry.is_none() { continue; } // Confirm: full string compare if !memcmp(data + offset, entry.pattern) { continue; } // Record match matches.push(Match { sig_id: entry.sig_id, offset, weight: entry.weight, }); // Accumulate group weight groups[entry.group] += entry.weight; // HSTR_WEIGHT @ 0x1097C6D0 }

Weight-Based Scoring

Multiple weak indicators combine to exceed a threshold -- reducing false positives

// Example: TrojanDropper:Script/Obfuse detection // Signature group with 6 patterns and threshold 80 Pattern Weight Found? "WScript.Shell" +30 YES "ActiveXObject" +20 no "eval(unescape" +40 YES "fromCharCode" +15 YES "XMLHTTP" +10 no "%TEMP%" +5 no ------ Total Weight: 85 Threshold: 80 Result: 85 >= 80 ==> MATCH // 3 of 6 patterns suffice
Weighted Not all patterns need to match -- strong signals outweigh weak ones
Polymorphic Handles variants that change some patterns but keep core behavior
Graduated Different confidence levels without requiring exact match sets

8 HSTR Caller Contexts

GetHSTRCallerId @ 0x1097F460 — Different matching profiles per pipeline stage

HSTR_CALLER_FILE @ 0x109812C0 — File scan context
HSTR_CALLER_EMS @ 0x109812D4 — Emulator context
HSTR_CALLER_HOOKWOW @ 0x109812F8 — WoW64 hook
HSTR_CALLER_SMS @ 0x10981308 — System monitor
HSTR_CALLER_CMDLINE @ 0x1098131C — Command line
HSTR_CALLER_TSKSCHED @ 0x1098133C — Task scheduler
HSTR_CALLER_BITS @ 0x10981350 — BITS job cmdline
HSTR_CALLER_UNKNOWN @ 0x109812AC — Default context

Why Caller Context Matters

Signature Selection Different caller IDs load different signature subsets from the VDM
Matching Parameters Weight thresholds and exhaustive mode can vary by context
False Positive Tuning Command line patterns that are malicious in BITS jobs may be benign in direct execution
// Exhaustive mode config "NID_ENABLE_HSTR_EXHAUSTIVE" @ 0x10980A50 // All matches found, not just first // Overlapping matches recorded

Per-Format Feature Extraction

BRUTE extracts format-specific features for ML classification and Lua analysis

BRUTE:PDF:Feature:

@ 0x10A54CC8
JavaScript presence/size
Embedded file detection
Suspicious action chains
Unusual filter chains
XFA form presence
Object stream anomalies

BRUTE:VBS:Feature:

@ 0x10A54D08
Shell execution patterns
File system access (FSO)
Network access (XMLHTTP)
Registry operations
Chr() density scoring
Obfuscation indicators

BRUTE:JS:Feature:

@ 0x10A54D1C
ActiveXObject patterns
eval() usage frequency
WScript.Shell calls
DOM manipulation
Base64 density
Encoded content detection
Features feed into SigTree ML + Lua scripts BRUTE:*:Feature:* attributes feed into SigTree decision trees (SS14) for !MTB/!ml detections, and Lua scripts (Stage 10) for complex logic

The HSTR Family Tree

BRUTE is one of 10+ HSTR matching variants, each targeting different content types

HSTR (base matching engine) | +-- BRUTE @ 0x10986D8C | Format-agnostic, raw bytes | +-- NSCRIPT_BRUTE @ 0x10986AB0 | Deobfuscated script content | +-- PEHSTR @ 0x109869C8 | +-- PEHSTR_EXT @ 0x10986618 | +-- PEHSTR_EXT2 @ 0x109863EC | +-- MACHOHSTR_EXT @ 0x109861D8 +-- DEXHSTR_EXT @ 0x10986430 +-- SWFHSTR_EXT @ 0x10986DF8 +-- MACROHSTR_EXT @ 0x109870A0 +-- JAVAHSTR_EXT @ 0x109870EC +-- ARHSTR_POSIX_EXT @ 0x10986F20 +-- ASCRIPTHSTR_EXT @ 0x109864D4

162 Signature Types Total

Hash-based KCRCE, LOCALHASH, BLOOM
Structure PESTATIC, PEBMPAT, IL_PATTERN
Behavior FOP, THREAD_X86, TUNNEL_X86
Script NSCRIPT_SP, NSCRIPT_BRUTE
Cloud FASTPATH_SDN, FASTPATH_TDN
Universal BRUTE -- the format-agnostic catch-all

BRUTE + Lua Integration

BRUTE match results are the primary data source for Lua-based complex detection

Data Flow

Stage 9: BRUTE
Match patterns A, C, F against raw data
↓ match results ↓
Stage 10: Lua
if match("A") and match("C") and not match("D") then DETECT
↓ detection ↓
THREAT: Complex multi-condition detection

What Lua Can Do With BRUTE Results

Check if specific BRUTE patterns matched
Get match counts for frequency analysis
Get match offsets for positional analysis
Combine BRUTE + attributes from other stages
Complex boolean logic beyond AAGG expressions
-- Lua script example if mp.bcrp_match("WScript.Shell") and mp.bcrp_match("eval(") and mp.getattribute( "Nscript:js_hasBigString") then mp.set_detection( "TrojanDropper:JS/Obfuse") end

NSCRIPT_BRUTE: Script Variant

Specialized BRUTE matching for deobfuscated script content

SIGNATURE_TYPE_BRUTE

@ 0x10986D8C
Input: Raw file bytes
Timing: After all transforms
Target: Any file format
Patterns: Binary patterns
Use: Polymorphic malware

SIGNATURE_TYPE_NSCRIPT_BRUTE

@ 0x10986AB0
Input: Deobfuscated script text
Timing: After NScript Stage 8
Target: Script content only
Patterns: Text/string patterns
Use: Obfuscated script malware
Obfuscated scripts have no stable byte patterns After NScript normalization, the underlying API calls and strings are exposed -- NSCRIPT_BRUTE matches against this stable, deobfuscated form

BRUTE Matching: Summary

0Total SIGNATURE_TYPE Entries
8HSTR Caller Contexts
10+Format-Specific HSTR Variants
3Feature Extraction Prefixes
O(n)Amortized Complexity
2BRUTE + NSCRIPT_BRUTE
WeightedMulti-Pattern Scoring
LuaPrimary Data for Stage 10

BRUTE matching is the format-agnostic safety net in the Defender pipeline. While format-specific engines provide fast targeted matching, BRUTE ensures that no content passes through without thorough raw-byte pattern analysis. Its weighted scoring and Lua integration enable detection of sophisticated polymorphic threats.

SIGNATURE_TYPE_BRUTE @ 0x10986D8C • HSTR_WEIGHT @ 0x1097C6D0 • 162 sig types • mpengine.dll