mirror of
https://github.com/SpecterOps/Nemesis
synced 2026-06-08 12:36:42 +00:00
Will Schroeder
a36be30b39
* Add k8s/k3d/k3s deployment with Helm, KEDA autoscaling, and connection pool fixes - Helm chart for full Nemesis deployment (Traefik, Dapr, KEDA, all services) - KEDA autoscaling for file-enrichment and document-conversion on RabbitMQ queue depth - Fix PostgreSQL connection exhaustion: configurable pool size (default 20, was 100) - Cap Dapr statestore connections (maxConns=10), increase postgres max_connections to 300 - Add Prometheus metrics and /system/pool-stats endpoint for connection pool monitoring - Background pool stats logger with pressure warnings - Fix Prometheus scrape targets to correct service ports - Helm-based Dapr install (v1.16.9) instead of CLI - k8s setup/deploy/build/verify/teardown scripts - Kubernetes deployment docs and references in CLAUDE.md, AGENTS.md, mkdocs * Add KEDA autoscaling for titus-scanner and dotnet-service * Add PgBouncer connection pooling, KEDA gotenberg CPU scaling, and tuning - Add PgBouncer deployment between services and PostgreSQL (transaction pooling) - Route all DB connections through pgbouncer instead of postgres directly - Add KEDA CPU-based autoscaling for gotenberg (1-3 replicas) - Mirror external images (pgbouncer) into k3d registry via build-and-push.sh - Reduce queue-based cooldowns to 60s, gotenberg stabilization to 30s - Add 180s terminationGracePeriodSeconds to gotenberg for in-flight conversions - Update docs with PgBouncer architecture, connection tuning, and troubleshooting * Add k3s support, unify Traefik on port 7443, rename scripts - Add setup-cluster-k3s.sh and teardown-cluster-k3s.sh for native k3s - Rename scripts with -k3d suffix to distinguish runtimes - Unify k3s Traefik to HTTPS-only on port 7443 (matching k3d) - Add k3d registry check for --build flag in deploy.sh - Fix verify.sh arithmetic under set -e with || true - Update docs with k3s quick start, comparison table, and teardown - Add image-sync workflow and pgbouncer registry override for values-dev * Add monitoring, Jupyter, and LLM optional stacks to Helm chart - Monitoring: Prometheus, Grafana (with dashboards), Loki, Promtail, Jaeger, OTEL Collector, Node Exporter, Postgres Exporter - Jupyter: notebook service with Hasura integration - LLM: Agents (Dapr-enabled), LiteLLM proxy, Phoenix observability - Add ingress routes, strip-prefix middleware, litellm database - Update deploy.sh, build script, README, and test connectivity checks * Add k3s --build support with auto-detection in deploy.sh - deploy.sh --build auto-detects k3d registry vs k3s and uses appropriate build script - Add build-and-load-k3s.sh for building and importing images into k3s containerd - Add values-dev-k3s.yaml with imagePullPolicy: Never (no pgbouncer registry override) - Fix jupyter Dockerfile context path in build-and-push-k3d.sh - Update docs for k3s local build support --------- Co-authored-by: Lee Chagolla-Christensen <lee@localhost>
Nemesis CLI
A command-line interface for the Nemesis platform that provides file submission, monitoring, and C2 connector functionality.
Purpose
This CLI tool serves as the primary interface for uploading files to Nemesis, monitoring directories for new files, and synchronizing data from C2 frameworks like Mythic, Cobalt Strike and Outflank.
Features
- File submission: Upload single files or entire directories to Nemesis
- Directory monitoring: Real-time monitoring of folders for new files
- C2 connectors: Synchronize data from Mythic, Cobalt Strike and Outflank C2 frameworks
- Stress testing: Load testing capabilities for the Nemesis API
- Module testing: Execute file enrichment modules standalone for development
Commands
submit
Upload files or directories to Nemesis for processing.
Key options:
-r, --recursive: Process subdirectories recursively-w, --workers: Number of concurrent upload threads (default: 10)--project: Project name for metadata (default: assess-test)--agent-id: Agent identifier for tracking uploads
monitor
Monitor a directory for new files and automatically submit them to Nemesis.
Key options:
--only-monitor: Skip existing files, only watch for new ones-w, --workers: Number of threads for initial submission
connect-mythic
Synchronize data between Mythic C2 framework and Nemesis.
Configuration:
- Uses
settings_mythic.yamlconfiguration file --showconfig: Display example configuration
connect-outflank
Ingest data from Outflank Stage1 C2 into Nemesis.
Configuration:
- Uses
settings_outflank.yamlconfiguration file --showconfig: Display example configuration
connect-cobaltstrike
Ingest data from Cobalt Strike into Nemesis.
Configuration:
- Uses
settings_cobaltstrike.yamlconfiguration file --showconfig: Display example configuration
Additional Tools
- stress_test: Load testing tool for API performance evaluation
- module_runner: Standalone execution of file enrichment modules for development and testing
Authentication
All commands support basic authentication with configurable username and password options (default: n/n).
Manually running with Python
- Navigate to the cli directory. Perform all the following steps from this directory.
cd Nemesis/projects/cli
- Install system dependencies required for building native packages:
# Debian/Ubuntu
sudo apt-get install -y libleveldb-dev
# macOS
brew install leveldb
- Install dependencies and run it:
uv sync
uv run python -m cli
Manually Building and Using with Docker
- Navigate to the cli directory. Perform all the following steps from this directory.
cd Nemesis/projects/cli
- Build the base images:
docker compose -f ../../compose.base.yaml build
- Build the nemesis-cli image:
docker build -t nemesis-cli --target prod --no-cache -f Dockerfile ../..
Validate --target arguments are prod or dev.
- Run the nemesis-cli container:
docker run --network host -v /:/data --rm nemesis-cli submit /data/etc/issue
Using Docker Compose
Pull the published production image and run it
- Navigate to the cli directory. Perform all the following steps from this directory.
cd Nemesis/projects/cli
- Pull the published production container and run it:
docker compose -f compose.yaml run --rm cli
Build and run the dev/production images
- Navigate to the cli directory. Perform all the following steps from this directory.
cd Nemesis/projects/cli
- Build the base images:
docker compose -f ../../compose.base.yaml build
- Build and run the dev or production containers.
To run the development container, run the following. This mounts CLI's code into container and uses the dev base image. It implicitly merges compose.yaml and compose.override.yaml.
docker compose run --rm cli
Alternatively, you can build the production image and run it with the following:
docker compose -f compose.yaml -f compose.prod.build.yaml run --rm cli
Using submit.sh (in dev)
Building the dev image
- Navigate to the cli directory. Perform all the following steps from this directory.
cd Nemesis/projects/cli
- Build the base images:
docker compose -f ../../compose.base.yaml build
- Build the nemesis-cli image:
docker build -t nemesis-cli --target dev --no-cache -f Dockerfile ../..
- Export NEMESIS_CLI_IMAGE
export NEMESIS_CLI_IMAGE=nemesis-cli:latest
- Run ./submit.sh as normal:
cd ../..
./tools/submit.sh --help