mirror of
https://github.com/SpecterOps/Nemesis
synced 2026-06-08 12:36:42 +00:00
Will Schroeder
79682c8aa9
* Conversion of NoseyParker scanner to Titus - Old `noseyparker_scanner` project removed, new `titus_scaner` Golang project built - Replaced `noseyparker` references with `titus` as appropriate - Bumped various built-in docker image versions (Dapr, Gotenberg, etc.) * Replace ZIP-only extraction with Titus library archive support Replace custom ZIP extraction code with Titus library's enum.ExtractText() to support scanning secrets inside zip, jar, war, ear, apk, ipa, xpi, crx, tar, tar.gz/tgz, and 7z archives. Document formats (xlsx, docx, pdf, etc.) are intentionally excluded since Nemesis handles those via document_conversion. - Add original_path to TitusInput so Titus can dispatch by file extension (MinIO stores files by UUID without extensions) - Add magic-byte fallback detection for ZIP, 7z, gzip, and tar - Preserve git-repo-in-ZIP detection as a special case before archive scan - Rename env vars: DECOMPRESS_ZIPS -> EXTRACT_ARCHIVES, MAX_EXTRACT_SIZE_MB -> EXTRACT_MAX_TOTAL_SIZE_MB, and add EXTRACT_MAX_FILE_SIZE_MB and EXTRACT_MAX_DEPTH - Add unit tests for archive detection and config loading * Integrated (optional) Titus secret validation - Integrated (optional) Titus secret validation that can be enabled * Added Titus rule ID display and ability to disable specific rules - Added Titus rule ID display and ability to disable specific built-in rules (by ID) * `titus_scanner` fixes - One scanner per concurrent file to prevent timeouts - Fix custom rule regexes to prevent timeouts - Exclude a few default run-away rules causing timeouts - Bumped cores for titus scanner to 2 * Enable Vectorscan/Hyperscan acceleration for titus_scanner Switch from the pure-Go regexp2 engine to the Vectorscan (Hyperscan) SIMD-accelerated engine, using a fork with parallelized regexp2 confirmation and content-based deduplication Key changes: - Dockerfile: install libvectorscan-dev/libvectorscan5, build with CGO_ENABLED=1 and -tags vectorscan - go.mod: replace praetorian-inc/titus with HarmJ0y/titus fork that parallelizes the regexp2 confirmation loop and uses content-based dedup (matching portable engine behavior) - Add bounded-concurrency semaphore in handler to prevent OOM from unbounded goroutines while still returning 200 to Dapr immediately - Add content-based deduplication, match count cap, snippet truncation, and payload size logging to prevent oversized publishes - Replace O(N*M) line/column computation with precomputed line index using binary search - Add GOMEMLIMIT and memory limit to container resources * Update README.md * remove `noseyparker_scanner` container * Add EVTX (Windows Event Log) enrichment module Parses .evtx files and extracts security-relevant data: - Markdown summary transform with event counts, timeline, unique accounts/IPs - Downloadable CSV transforms for account changes, group membership changes, explicit credential use (4648), process creation (4688), and task changes - Findings for high-signal events: audit log cleared, new services, admin group changes, account changes, explicit creds, scheduled tasks, PS script blocks - PowerShell 4104 script blocks reassembled and resubmitted to enrichment pipeline as child .ps1 files for full Titus/YARA scanning - Added de-duping for duplicate PS scripts carved from EVTXs (hash-based) * EVTX mods - Power timelining done for SYSTEM - Inbound/outbound auth for SECURITY - Filtering for default system accounts - CSV viewer now can download files, transforms for EVTX transformed to the viewer * Convert titus scanner to Dapr bulk subscribe for batch processing - Add bulk subscribe types to models (BulkMessagePayload, BulkResponse, etc.) - Replace async single-event handler with synchronous HandleBulkEvent - Process batches with bounded concurrency via semaphore + WaitGroup - Handle edge cases: duplicate entryId, empty objectID/entryId, panic recovery - Add BulkMaxMessages and BulkMaxAwaitDurationMs config with clamping - Set app-max-concurrency to 1 (parallelism within work batch via scanner pool) --------- Co-authored-by: Lee Chagolla-Christensen <lee@localhost>