mirror of
https://github.com/SpecterOps/Nemesis
synced 2026-06-08 12:36:42 +00:00
Will Schroeder
a36be30b39
* Add k8s/k3d/k3s deployment with Helm, KEDA autoscaling, and connection pool fixes - Helm chart for full Nemesis deployment (Traefik, Dapr, KEDA, all services) - KEDA autoscaling for file-enrichment and document-conversion on RabbitMQ queue depth - Fix PostgreSQL connection exhaustion: configurable pool size (default 20, was 100) - Cap Dapr statestore connections (maxConns=10), increase postgres max_connections to 300 - Add Prometheus metrics and /system/pool-stats endpoint for connection pool monitoring - Background pool stats logger with pressure warnings - Fix Prometheus scrape targets to correct service ports - Helm-based Dapr install (v1.16.9) instead of CLI - k8s setup/deploy/build/verify/teardown scripts - Kubernetes deployment docs and references in CLAUDE.md, AGENTS.md, mkdocs * Add KEDA autoscaling for titus-scanner and dotnet-service * Add PgBouncer connection pooling, KEDA gotenberg CPU scaling, and tuning - Add PgBouncer deployment between services and PostgreSQL (transaction pooling) - Route all DB connections through pgbouncer instead of postgres directly - Add KEDA CPU-based autoscaling for gotenberg (1-3 replicas) - Mirror external images (pgbouncer) into k3d registry via build-and-push.sh - Reduce queue-based cooldowns to 60s, gotenberg stabilization to 30s - Add 180s terminationGracePeriodSeconds to gotenberg for in-flight conversions - Update docs with PgBouncer architecture, connection tuning, and troubleshooting * Add k3s support, unify Traefik on port 7443, rename scripts - Add setup-cluster-k3s.sh and teardown-cluster-k3s.sh for native k3s - Rename scripts with -k3d suffix to distinguish runtimes - Unify k3s Traefik to HTTPS-only on port 7443 (matching k3d) - Add k3d registry check for --build flag in deploy.sh - Fix verify.sh arithmetic under set -e with || true - Update docs with k3s quick start, comparison table, and teardown - Add image-sync workflow and pgbouncer registry override for values-dev * Add monitoring, Jupyter, and LLM optional stacks to Helm chart - Monitoring: Prometheus, Grafana (with dashboards), Loki, Promtail, Jaeger, OTEL Collector, Node Exporter, Postgres Exporter - Jupyter: notebook service with Hasura integration - LLM: Agents (Dapr-enabled), LiteLLM proxy, Phoenix observability - Add ingress routes, strip-prefix middleware, litellm database - Update deploy.sh, build script, README, and test connectivity checks * Add k3s --build support with auto-detection in deploy.sh - deploy.sh --build auto-detects k3d registry vs k3s and uses appropriate build script - Add build-and-load-k3s.sh for building and importing images into k3s containerd - Add values-dev-k3s.yaml with imagePullPolicy: Never (no pgbouncer registry override) - Fix jupyter Dockerfile context path in build-and-push-k3d.sh - Update docs for k3s local build support --------- Co-authored-by: Lee Chagolla-Christensen <lee@localhost>
123 lines
6.1 KiB
Markdown
123 lines
6.1 KiB
Markdown
# Quickstart Guide
|
|
|
|
Here's a quickstart guide to setting up the Nemesis platform.
|
|
|
|
### Prerequisites
|
|
|
|
Ensure your machine meets the following requirements:
|
|
|
|
- **OS**: Linux (use Debian 12) or macOS
|
|
- **Processors**: 4 cores
|
|
- **Memory**: 12+ GB RAM
|
|
- **Disk Space**: 100 GB
|
|
- **Architecture**: x64 or Arm
|
|
|
|
- **Docker/Docker-Compose:**
|
|
- Docker version 28.0.0 or higher is recommended. See [Docker's installation instructions](https://docs.docker.com/engine/install/) for instructions on installing Docker. Running the Docker Engine on Linux or on OS X via Docker Desktop is recommended. If using Docker Desktop, ensure that the VM is configured with sufficient RAM/Disk/swap.
|
|
|
|
|
|
**NOTE:** for multi-language support for OCR/document processing, set the `TIKA_OCR_LANGUAGES` ENV var before launching with the [Tesseract language code](https://github.com/tesseract-ocr/tessdata):
|
|
```bash
|
|
$ export TIKA_OCR_LANGUAGES="eng chi_sim chi_tra jpn rus deu spa"
|
|
```
|
|
|
|
|
|
### Step 1: Clone the Nemesis Repository
|
|
```bash
|
|
git clone https://github.com/SpecterOps/Nemesis
|
|
cd Nemesis
|
|
```
|
|
|
|
### Step 2: Configuration
|
|
Create a `.env` file using the provided [env.example](https://github.com/SpecterOps/Nemesis/blob/main/env.example) as a template:
|
|
```bash
|
|
cp env.example .env
|
|
```
|
|
Then configure the values in the `.env` file with a text editor.
|
|
|
|
This file contains passwords and configuration that Nemesis uses. You should randomize these password values for your deployment!
|
|
|
|
Some Nemesis enrichments allow for (or require) configuration. See the [enrichment configuration](./enrichment_configuration.md) guide for details, for example to enable PII detection (which can be a performance hit).
|
|
|
|
**NOTE:** `NEMESIS_URL` is used to construct the appropriate absolute hyperlinks for findings and Apprise alerts. It does not affect the hosting of Nemesis itself. If using a hostname, a FQDN is preferred.
|
|
|
|
**NOTE:** for `APPRISE_URLs` to route user feedback to a specific channel use `?tag=feedback` as shown in the example .env file. Otherwise stock alerts will go to the first URL listed. See the [Alerting](./usage_guide.md#alerting) section of the Usage Guide for more information.
|
|
|
|
**NOTE:** To use your own SSL certificates, simply replace the `server.crt` and `server.key` files at `./infra/traefik/certs/` before launching Nemesis.
|
|
|
|
### Step 3: Start Nemesis
|
|
To start Nemesis's core services, run the `./tools/nemesis-ctl.sh` script:
|
|
|
|
```bash
|
|
./tools/nemesis-ctl.sh start prod
|
|
```
|
|
|
|
If you'd like to install the monitoring services, jupyter notebooks, and/or LLM agents use the associated optional command line arguments:
|
|
|
|
```bash
|
|
./tools/nemesis-ctl.sh start prod --monitoring --jupyter [--llm]
|
|
```
|
|
`nemesis-ctl.sh` effectively is a wrapper for `docker compose` commands and is in charge of pulling and starting the appropriate published Nemesis docker images. In general, we recommend people use `nemesis-ctl.sh` instead of manually invoking `docker compose`. For more complex deployment scenarios, see Nemesis's [Docker Compose documentation](docker_compose.md) to understand what `nemesis-ctl.sh` does underneath.
|
|
|
|
### Step 4: Access the Web Dashboard
|
|
|
|
In a web browser, open `https://localhost:7443/` (or the URL Nemesis is hosted on) to access the main Nemesis web interface. Use `n:n` for basic auth unless you specified users. Upon logging in, you will enter your username and project (this is saved in your browser cache and only needed once):
|
|
|
|
|
|
|
|
If needed, you can change these values by clicking the **Settings** tab on the lower left:
|
|
|
|
|
|
|
|
After entering your information, you will then be shown the main Nemesis dashboard with processed file statistics and enrichment workflow information:
|
|
|
|
|
|
|
|
### Step 5: Upload File for Analysis
|
|
|
|
To manually upload files into Nemesis, click on the "File Upload" link in the sidebar. On this page you can upload one or more files into Nemesis:
|
|
|
|
|
|
|
|
After uploading the files, click on the "Files" link in the sidebar. Once Nemesis processes the files, they will appear in the table:
|
|
|
|
|
|
|
|
Click on the table row to view the file's details:
|
|
|
|
|
|
See [Data Ingestion](./usage_guide.md#data-ingestion) for additional ways to ingest data into Nemesis besides manually uploading files through the web interface.
|
|
|
|
### Step 6: View Other Nemesis Services
|
|
|
|
Click on the "Help" button on the bottom left to view the additionally exposed Nemesis services. Each route listed is a hyperlink to the service. For logins, refer to the environment variables set.
|
|
|
|
**NOTE:** The monitoring services (Grafana, Jaeger, and Prometheus) will only be available if you started with them enabled (`--monitoring`).
|
|
|
|
**NOTE:** The /jupyter/ route will only be available if you started with it enabled (`--jupyter`).
|
|
|
|
|
|
|
|
### Alternative: Kubernetes (k3d) Deployment
|
|
|
|
For production-like environments with autoscaling, Nemesis can also be deployed to a local Kubernetes cluster using k3d. This provides KEDA-driven autoscaling, Dapr operator-managed sidecars, and Helm-based configuration. See the [Kubernetes deployment guide](kubernetes.md) for full instructions.
|
|
|
|
```bash
|
|
./k8s/scripts/setup-cluster.sh # Create cluster with Traefik, Dapr, KEDA
|
|
./k8s/scripts/deploy.sh install # Deploy Nemesis
|
|
./k8s/scripts/verify.sh # Verify deployment
|
|
```
|
|
|
|
### Step 7: Shutting Nemesis Down
|
|
|
|
To shutdown Nemesis, use the `nemesis-ctl.sh` script's `stop` or `clean` commands ***with the same arguments you used to start it***. For example, if you started it with monitoring, jupyter, or LLM support enabled, then run the following:
|
|
- To stop Nemesis containers:
|
|
```bash
|
|
./tools/nemesis-ctl.sh stop prod --monitoring --jupyter --llm
|
|
```
|
|
|
|
- To stop Nemesis containers and delete their associated volumes:
|
|
```bash
|
|
./tools/nemesis-ctl.sh clean prod --monitoring --jupyter --llm
|
|
```
|