mirror of
https://github.com/SpecterOps/Nemesis
synced 2026-06-08 12:36:42 +00:00
Lee Chagolla-Christensen
6ec0a3b61a
* upgrade to dapr postgresv2 statestore * actually make it v2 * dapr state table name, cleanup subscriptions/globals * proper exceptions * formatting/lint * Refactor workflow tracking and improve activity input handling - Extract workflow tracking logic into dedicated WorkflowTrackingService - Simplify activity signatures to accept specific parameters instead of generic dicts - Remove unused asyncio event loop references from enrichment modules - Update YaraRuleManager initialization and method names * re-added workflow tracking in the DB * update uvicorn prod options * enrichment work parallelism, convert queues from broadcast to task queues * Refactor pubsub and improve workflow parallelism - Split Dapr pubsub Dapr yaml components into topic-specific queues (alerting, dotnet, dpapi, files, noseyparker, workflow_monitor) - Update all Dapr volume mounts to reference new topic-specific pubsub components - Converted queues to task queues - Use YAML anchors to reduce duplication for file-enrichment replicas - Pass asyncpg pool to enrichment modules instead of creating connections - Add asyncpg_pool parameter throughout chromium and enrichment module analyzers - Update VSCode workspace (removed InspectAssembly, renamed dotnet_api to dotnet_service) - Added curl commands for Jaeger API to performance docs to help with perf troubleshooting - Created common.queues module to centralize pubsub/topic names (eases future refactoring) * worker mods * Workflow performance tuning, fix pubsub config, CLI arg changes - Fix pubsub deleteWhenUnused typo (deletedWhenUnused) - Add LOG_LEVEL environment variable support across services - CLI: Rename --repeat to --times, add --max-files option - Increase files pubsub prefetchCount from 25 to 50 - Add MAX_PARALLEL_WORKFLOWS configuration - Fix DotNetAssemblyAnalysis null handling with field validators - Update dashboard to show cumulative files/findings over time - Add RUST_LOG environment variable support to noseyparker - Update CHANGELOG for 2.1.4 release notes * Dapr 1.16.2 and use db transactions - Upgrade all Dapr containers from 1.16.1 to 1.16.2 - Reduce enrichment parallelism default from 25 to 5 workflows - Reduce healthcheck intervals from 10s to 5s for alerting and document conversion - Fixed DPAPI eventing to use new pubsubs - Refactor file_linking database operations to use atomic upserts and avoid deadlocks - Add WriteOnceViolationError handling in DPAPI masterkey analyzer - Wrap database operations in transactions for enrichment storage and plaintext indexing - Fix postgres notification handler closure variable capture * remove unused start_time * Scheduler persistence, workflow concurrency tuning, and config cleanup - Add volume for Dapr scheduler and init service - Add scheduler dependency to file enrichment service - Add async workflow client libraries - Format and cleanup compose.yaml (spacing, indentation, empty lines) * Migrate file_enrichment to async Dapr client and optimize Dockerfile - Use async DaprClient where possible in file_enrichment - Improve Dockerfile caching - Add asyncpg connection pool helper and fix typo in secret store name - Include VS Code debug configuration for document_conversion - Remove unused dapr_client from DpapiBlobAnalyzer - Clean up activity return types and better handle exceptions * Enrichment tracking for NoseyParker and logging cleanup - Add workflow_id to NoseyParkerInput and NoseyParkerOutput models - Remove workflow lookup query in noseyparker subscription handler - Adjust jaeger_perf_stats.sh output formatting and precision - Add type hints for async functions * noseyparker scanner perf, tracing for update_enrichment_results --------- Co-authored-by: Lee Chagolla-Christensen <lee@localhost>
Nemesis CLI
A command-line interface for the Nemesis platform that provides file submission, monitoring, and C2 connector functionality.
Purpose
This CLI tool serves as the primary interface for uploading files to Nemesis, monitoring directories for new files, and synchronizing data from C2 frameworks like Mythic and Outflank.
Features
- File submission: Upload single files or entire directories to Nemesis
- Directory monitoring: Real-time monitoring of folders for new files
- C2 connectors: Synchronize data from Mythic and Outflank C2 frameworks
- Stress testing: Load testing capabilities for the Nemesis API
- Module testing: Execute file enrichment modules standalone for development
Commands
submit
Upload files or directories to Nemesis for processing.
Key options:
-r, --recursive: Process subdirectories recursively-w, --workers: Number of concurrent upload threads (default: 10)--project: Project name for metadata (default: assess-test)--agent-id: Agent identifier for tracking uploads
monitor
Monitor a directory for new files and automatically submit them to Nemesis.
Key options:
--only-monitor: Skip existing files, only watch for new ones-w, --workers: Number of threads for initial submission
connect-mythic
Synchronize data between Mythic C2 framework and Nemesis.
Configuration:
- Uses
settings_mythic.yamlconfiguration file --showconfig: Display example configuration
connect-outflank
Ingest data from Outflank Stage1 C2 into Nemesis.
Configuration:
- Uses
settings_outflank.yamlconfiguration file --showconfig: Display example configuration
Additional Tools
- stress_test: Load testing tool for API performance evaluation
- module_runner: Standalone execution of file enrichment modules for development and testing
Authentication
All commands support basic authentication with configurable username and password options (default: n/n).
Manually running with Python
- Navigate to the cli directory. Perform all the following steps from this directory.
cd Nemesis/projects/cli
- Install dependencies and run it:
poetry install
poetry run python -m cli
Manually Building and Using with Docker
- Navigate to the cli directory. Perform all the following steps from this directory.
cd Nemesis/projects/cli
- Build the base images:
docker compose -f ../../compose.base.yaml build
- Build the nemesis-cli image:
docker build -t nemesis-cli --target prod --no-cache -f Dockerfile ../..
Validate --target arguments are prod or dev.
- Run the nemesis-cli container:
docker run --network host -v /:/data --rm nemesis-cli submit /data/etc/issue
Using Docker Compose
Pull the published production image and run it
- Navigate to the cli directory. Perform all the following steps from this directory.
cd Nemesis/projects/cli
- Pull the published production container and run it:
docker compose -f compose.yaml run --rm cli
Build and run the dev/production images
- Navigate to the cli directory. Perform all the following steps from this directory.
cd Nemesis/projects/cli
- Build the base images:
docker compose -f ../../compose.base.yaml build
- Build and run the dev or production containers.
To run the development container, run the following. This mounts CLI's code into container and uses the dev base image. It implicitly merges compose.yaml and compose.override.yaml.
docker compose run --rm cli
Alternatively, you can build the production image and run it with the following:
docker compose -f compose.yaml -f compose.prod.build.yaml run --rm cli
Using submit.sh (in dev)
Building the dev image
- Navigate to the cli directory. Perform all the following steps from this directory.
cd Nemesis/projects/cli
- Build the base images:
docker compose -f ../../compose.base.yaml build
- Build the nemesis-cli image:
docker build -t nemesis-cli --target dev --no-cache -f Dockerfile ../..
- Export NEMESIS_CLI_IMAGE
export NEMESIS_CLI_IMAGE=nemesis-cli:latest
- Run ./submit.sh as normal:
cd ../..
./tools/submit.sh --help