Will Schroeder
79682c8aa9
* Conversion of NoseyParker scanner to Titus - Old `noseyparker_scanner` project removed, new `titus_scaner` Golang project built - Replaced `noseyparker` references with `titus` as appropriate - Bumped various built-in docker image versions (Dapr, Gotenberg, etc.) * Replace ZIP-only extraction with Titus library archive support Replace custom ZIP extraction code with Titus library's enum.ExtractText() to support scanning secrets inside zip, jar, war, ear, apk, ipa, xpi, crx, tar, tar.gz/tgz, and 7z archives. Document formats (xlsx, docx, pdf, etc.) are intentionally excluded since Nemesis handles those via document_conversion. - Add original_path to TitusInput so Titus can dispatch by file extension (MinIO stores files by UUID without extensions) - Add magic-byte fallback detection for ZIP, 7z, gzip, and tar - Preserve git-repo-in-ZIP detection as a special case before archive scan - Rename env vars: DECOMPRESS_ZIPS -> EXTRACT_ARCHIVES, MAX_EXTRACT_SIZE_MB -> EXTRACT_MAX_TOTAL_SIZE_MB, and add EXTRACT_MAX_FILE_SIZE_MB and EXTRACT_MAX_DEPTH - Add unit tests for archive detection and config loading * Integrated (optional) Titus secret validation - Integrated (optional) Titus secret validation that can be enabled * Added Titus rule ID display and ability to disable specific rules - Added Titus rule ID display and ability to disable specific built-in rules (by ID) * `titus_scanner` fixes - One scanner per concurrent file to prevent timeouts - Fix custom rule regexes to prevent timeouts - Exclude a few default run-away rules causing timeouts - Bumped cores for titus scanner to 2 * Enable Vectorscan/Hyperscan acceleration for titus_scanner Switch from the pure-Go regexp2 engine to the Vectorscan (Hyperscan) SIMD-accelerated engine, using a fork with parallelized regexp2 confirmation and content-based deduplication Key changes: - Dockerfile: install libvectorscan-dev/libvectorscan5, build with CGO_ENABLED=1 and -tags vectorscan - go.mod: replace praetorian-inc/titus with HarmJ0y/titus fork that parallelizes the regexp2 confirmation loop and uses content-based dedup (matching portable engine behavior) - Add bounded-concurrency semaphore in handler to prevent OOM from unbounded goroutines while still returning 200 to Dapr immediately - Add content-based deduplication, match count cap, snippet truncation, and payload size logging to prevent oversized publishes - Replace O(N*M) line/column computation with precomputed line index using binary search - Add GOMEMLIMIT and memory limit to container resources * Update README.md * remove `noseyparker_scanner` container * Add EVTX (Windows Event Log) enrichment module Parses .evtx files and extracts security-relevant data: - Markdown summary transform with event counts, timeline, unique accounts/IPs - Downloadable CSV transforms for account changes, group membership changes, explicit credential use (4648), process creation (4688), and task changes - Findings for high-signal events: audit log cleared, new services, admin group changes, account changes, explicit creds, scheduled tasks, PS script blocks - PowerShell 4104 script blocks reassembled and resubmitted to enrichment pipeline as child .ps1 files for full Titus/YARA scanning - Added de-duping for duplicate PS scripts carved from EVTXs (hash-based) * EVTX mods - Power timelining done for SYSTEM - Inbound/outbound auth for SECURITY - Filtering for default system accounts - CSV viewer now can download files, transforms for EVTX transformed to the viewer * Convert titus scanner to Dapr bulk subscribe for batch processing - Add bulk subscribe types to models (BulkMessagePayload, BulkResponse, etc.) - Replace async single-event handler with synchronous HandleBulkEvent - Process batches with bounded concurrency via semaphore + WaitGroup - Handle edge cases: duplicate entryId, empty objectID/entryId, panic recovery - Add BulkMaxMessages and BulkMaxAwaitDurationMs config with clamping - Set app-max-concurrency to 1 (parallelism within work batch via scanner pool) --------- Co-authored-by: Lee Chagolla-Christensen <lee@localhost>
8.9 KiB
Nemesis Performance Tuning
This document details different ways to monitor and tune Nemesis's performance. Nemesis performs differently depending on a variety of factors, including the host's architecture and resources (particularly CPU, RAM, and disk speed) and the workload (e.g. the # of files and imbalances in the number of documents, .NET assemblies, source code, etc.).
If workflows begin to fail, or you are experiencing major performance issues, there are a few tunable settings that can help. Alternatively, if your performance is fine already and you want to potentially increase performance more or potentially reduce CPU/RAM usage (to save $$$), you can adjust these values. This document primarily focuses on increasing performance, but you can of course adjust the settings down to decrease resources.
Hardware Resourcing
The first thing to check is if Nemesis has enough hardware resources.
CPU
Under load, monitor CPU usage (e.g. with top/htop or the "Node Exporter" Grafana dashboard (if monitoring is enabled in Nemesis). If all cores are at lower utilization or not maxed out, continue following through this guide. Otherwise, you'll need to increase CPU resources for Nemesis since Nemesis is primarily CPU bound.
RAM
Under load, monitor RAM usage (e.g. with top/htop, free -h, or the "Node Exporter" Grafana dashboard if monitoring is enabled in Nemesis). Ensure that all memory is not being used; otherwise, you will need to increase RAM.
Note that Nemesis will buffer/cache memory if it can. Minio in particular will use any available RAM to cache file data in RAM. This memory is reclaimable, and therefore is still useable by other services/applications. We recommend having at least 1Gb of cache memory available. More may improve performance, but for the most part Nemesis is CPU bound, not RAM bound. You can apply docker compose memory limits to specific services if you want to constrain how much RAM minio consumes.
Disk
The requirements will vary widely here depending on your workload size. A general rule of thumb is 3x the size of all the files being uploaded. Use SSDs if possible.
Analyzing Your Workload
Analyzing Queues
Normally people realize Nemesis isn't going fast enough after uploading a bunch of files and it taking forever to process. Usually this is indicative that files get queued up for processing, but aren't processed fast enough. You can confirm this by analyzing the message queues in Nemesis/RabbitMQ.
In RabbitMQ, Ready counts signify messages waiting to be processed and the delivery / get rates (messages per second) will give you an idea of the processing speed. The following table maps the service to queue mappings:
| Docker Service | Queue Name | Description |
|---|---|---|
| file_enrichment | files-new_file | Uploaded files that haven't begun processing |
| document_conversion | files-document_conversion_input | Files waiting to go through document_conversion (strings, text extraction, PDF conversion) |
| dotnet_service | dotnet-dotnet_input | Files waiting for .NET decompilation and inspect assembly |
| titus-scanner | titus-titus_input | Files waiting to be scanned by titus |
If the queue message rates are too slow, you can adjust some settings to try and increase performance. The following sections detail the best bang-for-the-buck service-specific adjustments you can make.
file_enrichment
Every uploaded file is first placed on the files-new_file queue. The file_enrichment service consumes files from the queue and processes each one with the applicable enrichment modules. To improve file_enrichment performance, analyze its CPU usage with docker compose stats file-enrichment or in the "Docker Monitoring" dashboard in Grafana.
The first thing to tune is making sure file_enrichment is efficiently using a single core (currently, the file_enrichment service does not take full advantage of parallelism). Good utilization will look like ~90-110% CPU usage. i.e. the worker thread is taking full advantage of a single core. If CPU utilization is low, increase the number of workers with the ENRICHMENT_MAX_PARALLEL_WORKFLOWS environment variable (default is 5, meaning 5 workers). You'll also want to make sure this isn't set too high, causing workers to compete for CPU amongst themselves. If you increase to ~100 workers, then you'll also need to adjust Dapr's RabbitMQ prefetchCount count in file.yaml.
If additional cores are available, you can scale the file_enrichment container by adding replicas. Do this by modifying both the compose.yaml and compose.prod.yaml files, uncommenting the disabled file-enrichment-### placeholder replicas therein. Feel free to add more by following the same pattern, if wanted.
document_conversion
Every file is added to the files-document_conversion_input queue. The document_conversion service consumes files from the queue and extracts text, runs strings on the file, and converts documents to PDFs. To improve document_conversion performance, analyze its CPU usage with docker compose stats document-conversion or in the "Docker Monitoring" dashboard in Grafana. The document_conversion service can take full advantage of parallelism (so adding replicas is not necessary since a single instance can utilize multiple cores). However, the compose.yaml has resource limits that restrict the document-conversion service to 2 cores by default (adjust it if needed). In addition, you can adjust the DOCUMENTCONVERSION_MAX_PARALLEL_WORKFLOWS environment variable to adjust the number of workers (2 workers by default).
titus-scanner
Every text file is added to the titus-titus_input queue. The titus-scanner service consumes files from the queue and scans them with titus. To improve titus-scanner performance, analyze its CPU usage with docker compose stats titus-scanner or in the "Docker Monitoring" dashboard in Grafana. The titus-scanner service can take full advantage of parallelism (so adding replicas is not necessary since a single instance can utilize multiple cores). However, the compose.yaml has resource limits that restrict the titus-scanner service to 2 cores by default (adjust it if needed). In addition, you can adjust the TITUS_MAX_CONCURRENT_FILES environment variable to adjust the number of workers (2 workers by default). It's recommended to set TITUS_MAX_CONCURRENT_FILES to the same number of cores given to the service.
Dapr Scaling
Nemesis uses Dapr Workflows to build durable and reliable enrichment pipelines. Underneath, the workflows are managed by Dapr's scheduler service, which shares usage of the Postgres database with Nemesis.
You may need to scale the Dapr infrastructure if you considerably increase the performance of the file_enrichment and/or document_conversion services. Scaling Dapr is beyond the scope of this document, but here's some indicators when you may need to:
- Significant sustained CPU usage (> 80%-90%) by the scheduler container and/or Postgres container.
- Workflows begin failing frequently.
- You notice frequent activity failures/retries in Jaeger traces.
If this is the case, first try increasing the number of scheduler instances (example). Dapr does not support more than 3 scheduler instances unless you migrate to using an external etcd store. If Postgres begins to be the bottleneck, you may consider using a separate Postgres instance to store Dapr state.
Additional resources: