mirror of
https://github.com/SpecterOps/Nemesis
synced 2026-06-08 12:36:42 +00:00
Lee Chagolla-Christensen
6ec0a3b61a
* upgrade to dapr postgresv2 statestore * actually make it v2 * dapr state table name, cleanup subscriptions/globals * proper exceptions * formatting/lint * Refactor workflow tracking and improve activity input handling - Extract workflow tracking logic into dedicated WorkflowTrackingService - Simplify activity signatures to accept specific parameters instead of generic dicts - Remove unused asyncio event loop references from enrichment modules - Update YaraRuleManager initialization and method names * re-added workflow tracking in the DB * update uvicorn prod options * enrichment work parallelism, convert queues from broadcast to task queues * Refactor pubsub and improve workflow parallelism - Split Dapr pubsub Dapr yaml components into topic-specific queues (alerting, dotnet, dpapi, files, noseyparker, workflow_monitor) - Update all Dapr volume mounts to reference new topic-specific pubsub components - Converted queues to task queues - Use YAML anchors to reduce duplication for file-enrichment replicas - Pass asyncpg pool to enrichment modules instead of creating connections - Add asyncpg_pool parameter throughout chromium and enrichment module analyzers - Update VSCode workspace (removed InspectAssembly, renamed dotnet_api to dotnet_service) - Added curl commands for Jaeger API to performance docs to help with perf troubleshooting - Created common.queues module to centralize pubsub/topic names (eases future refactoring) * worker mods * Workflow performance tuning, fix pubsub config, CLI arg changes - Fix pubsub deleteWhenUnused typo (deletedWhenUnused) - Add LOG_LEVEL environment variable support across services - CLI: Rename --repeat to --times, add --max-files option - Increase files pubsub prefetchCount from 25 to 50 - Add MAX_PARALLEL_WORKFLOWS configuration - Fix DotNetAssemblyAnalysis null handling with field validators - Update dashboard to show cumulative files/findings over time - Add RUST_LOG environment variable support to noseyparker - Update CHANGELOG for 2.1.4 release notes * Dapr 1.16.2 and use db transactions - Upgrade all Dapr containers from 1.16.1 to 1.16.2 - Reduce enrichment parallelism default from 25 to 5 workflows - Reduce healthcheck intervals from 10s to 5s for alerting and document conversion - Fixed DPAPI eventing to use new pubsubs - Refactor file_linking database operations to use atomic upserts and avoid deadlocks - Add WriteOnceViolationError handling in DPAPI masterkey analyzer - Wrap database operations in transactions for enrichment storage and plaintext indexing - Fix postgres notification handler closure variable capture * remove unused start_time * Scheduler persistence, workflow concurrency tuning, and config cleanup - Add volume for Dapr scheduler and init service - Add scheduler dependency to file enrichment service - Add async workflow client libraries - Format and cleanup compose.yaml (spacing, indentation, empty lines) * Migrate file_enrichment to async Dapr client and optimize Dockerfile - Use async DaprClient where possible in file_enrichment - Improve Dockerfile caching - Add asyncpg connection pool helper and fix typo in secret store name - Include VS Code debug configuration for document_conversion - Remove unused dapr_client from DpapiBlobAnalyzer - Clean up activity return types and better handle exceptions * Enrichment tracking for NoseyParker and logging cleanup - Add workflow_id to NoseyParkerInput and NoseyParkerOutput models - Remove workflow lookup query in noseyparker subscription handler - Adjust jaeger_perf_stats.sh output formatting and precision - Add type hints for async functions * noseyparker scanner perf, tracing for update_enrichment_results --------- Co-authored-by: Lee Chagolla-Christensen <lee@localhost>
1.6 KiB
1.6 KiB
Performance Tuning
Nemesis may perform differently depending on the system architecture and resources, specifically RAM and the number of CPUs.
If workflows begin to fail, or you are experiencing major performance issues (as diagnosed by the Troubleshooting document) there are a few tunable parameters that can help. Alternatively, if your performance is fine and you want to potentially increase performance, you can increase these values. Most/all of these values involve altering behaviors for the file-enrichment service.
Useful Prometheus Metrics
Minio
minio_cluster_usage_objects_count{}
Jaeger
See how long a particular activity takes:
curl -sk --user 'n:n' "https://localhost:7443/jaeger/api/traces?service=file-enrichment&operation=activity%7C%7Crun_enrichment_modules&limit=2000" | jq -r '
[
.data[]
.spans[]
| select(.operationName == "activity||run_enrichment_modules")
| .duration
] as $durs
| {
count: ($durs | length),
min_ms: ($durs | min / 1000),
max_ms: ($durs | max / 1000),
avg_ms: (($durs | add / ($durs | length)) / 1000)
}
'
TODO: Need to document these
- Adjust enrichment workers
- Add some metrics around throughput and RAM/CPU consumption
- Adjust dapr workflow/activity concurrency settings
- How to tune your system to the right settings
- Seeing gaps between activities in Jaeger
- Useful Grafana/prometheus metrics
- Scaling the Dapr scheduler:
- How to determine if scheduler is slow down
- Creating scheduler replicas
- Disk performance recommendations (Fast/ramdisk suggestion)
- Use separate DB instance from app