* Conversion of NoseyParker scanner to Titus
- Old `noseyparker_scanner` project removed, new `titus_scaner` Golang project built
- Replaced `noseyparker` references with `titus` as appropriate
- Bumped various built-in docker image versions (Dapr, Gotenberg, etc.)
* Replace ZIP-only extraction with Titus library archive support
Replace custom ZIP extraction code with Titus library's enum.ExtractText()
to support scanning secrets inside zip, jar, war, ear, apk, ipa, xpi, crx,
tar, tar.gz/tgz, and 7z archives. Document formats (xlsx, docx, pdf, etc.)
are intentionally excluded since Nemesis handles those via document_conversion.
- Add original_path to TitusInput so Titus can dispatch by file extension
(MinIO stores files by UUID without extensions)
- Add magic-byte fallback detection for ZIP, 7z, gzip, and tar
- Preserve git-repo-in-ZIP detection as a special case before archive scan
- Rename env vars: DECOMPRESS_ZIPS -> EXTRACT_ARCHIVES,
MAX_EXTRACT_SIZE_MB -> EXTRACT_MAX_TOTAL_SIZE_MB, and add
EXTRACT_MAX_FILE_SIZE_MB and EXTRACT_MAX_DEPTH
- Add unit tests for archive detection and config loading
* Integrated (optional) Titus secret validation
- Integrated (optional) Titus secret validation that can be enabled
* Added Titus rule ID display and ability to disable specific rules
- Added Titus rule ID display and ability to disable specific built-in rules (by ID)
* `titus_scanner` fixes
- One scanner per concurrent file to prevent timeouts
- Fix custom rule regexes to prevent timeouts
- Exclude a few default run-away rules causing timeouts
- Bumped cores for titus scanner to 2
* Enable Vectorscan/Hyperscan acceleration for titus_scanner
Switch from the pure-Go regexp2 engine to the Vectorscan (Hyperscan)
SIMD-accelerated engine, using a fork with parallelized regexp2
confirmation and content-based deduplication
Key changes:
- Dockerfile: install libvectorscan-dev/libvectorscan5, build with
CGO_ENABLED=1 and -tags vectorscan
- go.mod: replace praetorian-inc/titus with HarmJ0y/titus fork that
parallelizes the regexp2 confirmation loop and uses content-based
dedup (matching portable engine behavior)
- Add bounded-concurrency semaphore in handler to prevent OOM from
unbounded goroutines while still returning 200 to Dapr immediately
- Add content-based deduplication, match count cap, snippet truncation,
and payload size logging to prevent oversized publishes
- Replace O(N*M) line/column computation with precomputed line index
using binary search
- Add GOMEMLIMIT and memory limit to container resources
* Update README.md
* remove `noseyparker_scanner` container
* Add EVTX (Windows Event Log) enrichment module
Parses .evtx files and extracts security-relevant data:
- Markdown summary transform with event counts, timeline, unique accounts/IPs
- Downloadable CSV transforms for account changes, group membership changes,
explicit credential use (4648), process creation (4688), and task changes
- Findings for high-signal events: audit log cleared, new services, admin group
changes, account changes, explicit creds, scheduled tasks, PS script blocks
- PowerShell 4104 script blocks reassembled and resubmitted to enrichment
pipeline as child .ps1 files for full Titus/YARA scanning
- Added de-duping for duplicate PS scripts carved from EVTXs (hash-based)
* EVTX mods
- Power timelining done for SYSTEM
- Inbound/outbound auth for SECURITY
- Filtering for default system accounts
- CSV viewer now can download files, transforms for EVTX transformed to the viewer
* Convert titus scanner to Dapr bulk subscribe for batch processing
- Add bulk subscribe types to models (BulkMessagePayload, BulkResponse, etc.)
- Replace async single-event handler with synchronous HandleBulkEvent
- Process batches with bounded concurrency via semaphore + WaitGroup
- Handle edge cases: duplicate entryId, empty objectID/entryId, panic recovery
- Add BulkMaxMessages and BulkMaxAwaitDurationMs config with clamping
- Set app-max-concurrency to 1 (parallelism within work batch via scanner pool)
---------
Co-authored-by: Lee Chagolla-Christensen <lee@localhost>
* upgrade to dapr postgresv2 statestore
* actually make it v2
* dapr state table name, cleanup subscriptions/globals
* proper exceptions
* formatting/lint
* Refactor workflow tracking and improve activity input handling
- Extract workflow tracking logic into dedicated WorkflowTrackingService
- Simplify activity signatures to accept specific parameters instead of generic dicts
- Remove unused asyncio event loop references from enrichment modules
- Update YaraRuleManager initialization and method names
* re-added workflow tracking in the DB
* update uvicorn prod options
* enrichment work parallelism, convert queues from broadcast to task queues
* Refactor pubsub and improve workflow parallelism
- Split Dapr pubsub Dapr yaml components into topic-specific queues (alerting, dotnet, dpapi, files, noseyparker, workflow_monitor)
- Update all Dapr volume mounts to reference new topic-specific pubsub components
- Converted queues to task queues
- Use YAML anchors to reduce duplication for file-enrichment replicas
- Pass asyncpg pool to enrichment modules instead of creating connections
- Add asyncpg_pool parameter throughout chromium and enrichment module analyzers
- Update VSCode workspace (removed InspectAssembly, renamed dotnet_api to dotnet_service)
- Added curl commands for Jaeger API to performance docs to help with perf troubleshooting
- Created common.queues module to centralize pubsub/topic names (eases
future refactoring)
* worker mods
* Workflow performance tuning, fix pubsub config, CLI arg changes
- Fix pubsub deleteWhenUnused typo (deletedWhenUnused)
- Add LOG_LEVEL environment variable support across services
- CLI: Rename --repeat to --times, add --max-files option
- Increase files pubsub prefetchCount from 25 to 50
- Add MAX_PARALLEL_WORKFLOWS configuration
- Fix DotNetAssemblyAnalysis null handling with field validators
- Update dashboard to show cumulative files/findings over time
- Add RUST_LOG environment variable support to noseyparker
- Update CHANGELOG for 2.1.4 release notes
* Dapr 1.16.2 and use db transactions
- Upgrade all Dapr containers from 1.16.1 to 1.16.2
- Reduce enrichment parallelism default from 25 to 5 workflows
- Reduce healthcheck intervals from 10s to 5s for alerting and document conversion
- Fixed DPAPI eventing to use new pubsubs
- Refactor file_linking database operations to use atomic upserts and avoid deadlocks
- Add WriteOnceViolationError handling in DPAPI masterkey analyzer
- Wrap database operations in transactions for enrichment storage and plaintext indexing
- Fix postgres notification handler closure variable capture
* remove unused start_time
* Scheduler persistence, workflow concurrency tuning, and config cleanup
- Add volume for Dapr scheduler and init service
- Add scheduler dependency to file enrichment service
- Add async workflow client libraries
- Format and cleanup compose.yaml (spacing, indentation, empty lines)
* Migrate file_enrichment to async Dapr client and optimize Dockerfile
- Use async DaprClient where possible in file_enrichment
- Improve Dockerfile caching
- Add asyncpg connection pool helper and fix typo in secret store name
- Include VS Code debug configuration for document_conversion
- Remove unused dapr_client from DpapiBlobAnalyzer
- Clean up activity return types and better handle exceptions
* Enrichment tracking for NoseyParker and logging cleanup
- Add workflow_id to NoseyParkerInput and NoseyParkerOutput models
- Remove workflow lookup query in noseyparker subscription handler
- Adjust jaeger_perf_stats.sh output formatting and precision
- Add type hints for async functions
* noseyparker scanner perf, tracing for update_enrichment_results
---------
Co-authored-by: Lee Chagolla-Christensen <lee@localhost>
Lee Chagolla-Christensen