* Add k8s/k3d/k3s deployment with Helm, KEDA autoscaling, and connection pool fixes - Helm chart for full Nemesis deployment (Traefik, Dapr, KEDA, all services) - KEDA autoscaling for file-enrichment and document-conversion on RabbitMQ queue depth - Fix PostgreSQL connection exhaustion: configurable pool size (default 20, was 100) - Cap Dapr statestore connections (maxConns=10), increase postgres max_connections to 300 - Add Prometheus metrics and /system/pool-stats endpoint for connection pool monitoring - Background pool stats logger with pressure warnings - Fix Prometheus scrape targets to correct service ports - Helm-based Dapr install (v1.16.9) instead of CLI - k8s setup/deploy/build/verify/teardown scripts - Kubernetes deployment docs and references in CLAUDE.md, AGENTS.md, mkdocs * Add KEDA autoscaling for titus-scanner and dotnet-service * Add PgBouncer connection pooling, KEDA gotenberg CPU scaling, and tuning - Add PgBouncer deployment between services and PostgreSQL (transaction pooling) - Route all DB connections through pgbouncer instead of postgres directly - Add KEDA CPU-based autoscaling for gotenberg (1-3 replicas) - Mirror external images (pgbouncer) into k3d registry via build-and-push.sh - Reduce queue-based cooldowns to 60s, gotenberg stabilization to 30s - Add 180s terminationGracePeriodSeconds to gotenberg for in-flight conversions - Update docs with PgBouncer architecture, connection tuning, and troubleshooting * Add k3s support, unify Traefik on port 7443, rename scripts - Add setup-cluster-k3s.sh and teardown-cluster-k3s.sh for native k3s - Rename scripts with -k3d suffix to distinguish runtimes - Unify k3s Traefik to HTTPS-only on port 7443 (matching k3d) - Add k3d registry check for --build flag in deploy.sh - Fix verify.sh arithmetic under set -e with || true - Update docs with k3s quick start, comparison table, and teardown - Add image-sync workflow and pgbouncer registry override for values-dev * Add monitoring, Jupyter, and LLM optional stacks to Helm chart - Monitoring: Prometheus, Grafana (with dashboards), Loki, Promtail, Jaeger, OTEL Collector, Node Exporter, Postgres Exporter - Jupyter: notebook service with Hasura integration - LLM: Agents (Dapr-enabled), LiteLLM proxy, Phoenix observability - Add ingress routes, strip-prefix middleware, litellm database - Update deploy.sh, build script, README, and test connectivity checks * Add k3s --build support with auto-detection in deploy.sh - deploy.sh --build auto-detects k3d registry vs k3s and uses appropriate build script - Add build-and-load-k3s.sh for building and importing images into k3s containerd - Add values-dev-k3s.yaml with imagePullPolicy: Never (no pgbouncer registry override) - Fix jupyter Dockerfile context path in build-and-push-k3d.sh - Update docs for k3s local build support --------- Co-authored-by: Lee Chagolla-Christensen <lee@localhost>
GitHub Actions for Nemesis
This directory contains GitHub Action workflows for automating tasks related to the Nemesis project.
Docker Build and Publish Workflow
The docker-build.yml workflow automatically builds and publishes Docker images for all Nemesis services to the GitHub Container Registry (ghcr.io).
Workflow Trigger
The workflow runs on:
- Push to the
mainbranch (when specific paths are modified) - Pull requests targeting the
mainbranch (when specific paths are modified) - Manual triggers via the GitHub Actions UI (workflow_dispatch)
What It Does
-
Base Images Build Job:
- Builds the Python base images (dev and prod) and the InspectAssembly image
- Pushes them to the GitHub Container Registry
- Passes the image tags to the next job
-
Service Images Build Job:
- Uses a matrix strategy to build all service images in parallel
- Builds both development and production targets for each service
- References the base images built in the first job
- Pushes all images to the GitHub Container Registry
Image Tags
Each image is tagged with:
- The short SHA of the commit
- The branch name
latest(only for the default branch)
Development images are additionally tagged with -dev suffix.
How to Use the Published Images
In docker compose
To use these images in your docker compose file, update your service definitions:
services:
web-api:
image: ghcr.io/your-org/nemesis/web-api:latest
# For development images
# image: ghcr.io/your-org/nemesis/web-api:latest-dev
In a Production Environment
For production deployments, reference specific versions by commit SHA for stability:
services:
web-api:
image: ghcr.io/your-org/nemesis/web-api:sha-abc123
Required Repository Secrets
For this workflow to function properly, ensure:
- Your repository has appropriate permissions to write packages
- GitHub Actions has permission to create and push container images
No additional secrets are needed as the workflow uses the built-in GITHUB_TOKEN.
Customization
To customize the workflow:
- Change the registry by modifying the
REGISTRYenv variable - Update the image naming prefix in the
IMAGE_PREFIXenv variable - Add or remove services in the matrix configuration
Documentation Build and Publish Workflow
The docs.yml workflow automatically builds and publishes the Nemesis project documentation to GitHub Pages whenever documentation files are updated.
Workflow Trigger
The workflow runs on:
- Push to the
mainbranch when specific documentation-related files are modified:- Files in the
docs/directory - The
mkdocs.ymlconfiguration file - The workflow file itself (
.github/workflows/docs.yml)
- Files in the
Environment and Permissions
The workflow runs in the github-pages environment with the following permissions:
pages: write: Allows publishing to GitHub Pagesid-token: write: Enables secure deploymentcontents: read: Provides access to repository content
How Documentation is Generated
The workflow uses MkDocs, a popular Python-based documentation generator:
- Documentation source files are written in Markdown
- The
mkdocs.ymlfile configures the structure and appearance - MkDocs processes these files to create a static website
- The generated site is deployed to GitHub Pages
Accessing the Published Documentation
After a successful workflow run, the documentation is available at:
https://<organization-name>.github.io/<repository-name>/
Code Vulnerability Scan Workflow
The vuln-scan.yml workflow automatically scans the Nemesis codebase for security vulnerabilities using Aqua Security's Trivy scanner.
Workflow Trigger
The workflow runs on:
- Push to the
mainbranch - Pull requests targeting the
mainbranch (excluding changes to documentation files)