Files
Will Schroeder a36be30b39 K8s k3d/k3s support (#106)
* Add k8s/k3d/k3s deployment with Helm, KEDA autoscaling, and connection pool fixes

- Helm chart for full Nemesis deployment (Traefik, Dapr, KEDA, all services)
- KEDA autoscaling for file-enrichment and document-conversion on RabbitMQ queue depth
- Fix PostgreSQL connection exhaustion: configurable pool size (default 20, was 100)
- Cap Dapr statestore connections (maxConns=10), increase postgres max_connections to 300
- Add Prometheus metrics and /system/pool-stats endpoint for connection pool monitoring
- Background pool stats logger with pressure warnings
- Fix Prometheus scrape targets to correct service ports
- Helm-based Dapr install (v1.16.9) instead of CLI
- k8s setup/deploy/build/verify/teardown scripts
- Kubernetes deployment docs and references in CLAUDE.md, AGENTS.md, mkdocs

* Add KEDA autoscaling for titus-scanner and dotnet-service

* Add PgBouncer connection pooling, KEDA gotenberg CPU scaling, and tuning

- Add PgBouncer deployment between services and PostgreSQL (transaction pooling)
- Route all DB connections through pgbouncer instead of postgres directly
- Add KEDA CPU-based autoscaling for gotenberg (1-3 replicas)
- Mirror external images (pgbouncer) into k3d registry via build-and-push.sh
- Reduce queue-based cooldowns to 60s, gotenberg stabilization to 30s
- Add 180s terminationGracePeriodSeconds to gotenberg for in-flight conversions
- Update docs with PgBouncer architecture, connection tuning, and troubleshooting

* Add k3s support, unify Traefik on port 7443, rename scripts

- Add setup-cluster-k3s.sh and teardown-cluster-k3s.sh for native k3s
- Rename scripts with -k3d suffix to distinguish runtimes
- Unify k3s Traefik to HTTPS-only on port 7443 (matching k3d)
- Add k3d registry check for --build flag in deploy.sh
- Fix verify.sh arithmetic under set -e with || true
- Update docs with k3s quick start, comparison table, and teardown
- Add image-sync workflow and pgbouncer registry override for values-dev

* Add monitoring, Jupyter, and LLM optional stacks to Helm chart

- Monitoring: Prometheus, Grafana (with dashboards), Loki, Promtail,
  Jaeger, OTEL Collector, Node Exporter, Postgres Exporter
- Jupyter: notebook service with Hasura integration
- LLM: Agents (Dapr-enabled), LiteLLM proxy, Phoenix observability
- Add ingress routes, strip-prefix middleware, litellm database
- Update deploy.sh, build script, README, and test connectivity checks

* Add k3s --build support with auto-detection in deploy.sh

- deploy.sh --build auto-detects k3d registry vs k3s and uses appropriate build script
- Add build-and-load-k3s.sh for building and importing images into k3s containerd
- Add values-dev-k3s.yaml with imagePullPolicy: Never (no pgbouncer registry override)
- Fix jupyter Dockerfile context path in build-and-push-k3d.sh
- Update docs for k3s local build support

---------

Co-authored-by: Lee Chagolla-Christensen <lee@localhost>
2026-03-05 10:29:03 -08:00
..
2026-01-14 17:32:09 -08:00
2026-01-14 17:32:09 -08:00
2026-03-05 10:29:03 -08:00
2025-06-13 11:33:07 +02:00

GitHub Actions for Nemesis

This directory contains GitHub Action workflows for automating tasks related to the Nemesis project.

Docker Build and Publish Workflow

The docker-build.yml workflow automatically builds and publishes Docker images for all Nemesis services to the GitHub Container Registry (ghcr.io).

Workflow Trigger

The workflow runs on:

  • Push to the main branch (when specific paths are modified)
  • Pull requests targeting the main branch (when specific paths are modified)
  • Manual triggers via the GitHub Actions UI (workflow_dispatch)

What It Does

  1. Base Images Build Job:

    • Builds the Python base images (dev and prod) and the InspectAssembly image
    • Pushes them to the GitHub Container Registry
    • Passes the image tags to the next job
  2. Service Images Build Job:

    • Uses a matrix strategy to build all service images in parallel
    • Builds both development and production targets for each service
    • References the base images built in the first job
    • Pushes all images to the GitHub Container Registry

Image Tags

Each image is tagged with:

  • The short SHA of the commit
  • The branch name
  • latest (only for the default branch)

Development images are additionally tagged with -dev suffix.

How to Use the Published Images

In docker compose

To use these images in your docker compose file, update your service definitions:

services:
  web-api:
    image: ghcr.io/your-org/nemesis/web-api:latest
    # For development images
    # image: ghcr.io/your-org/nemesis/web-api:latest-dev

In a Production Environment

For production deployments, reference specific versions by commit SHA for stability:

services:
  web-api:
    image: ghcr.io/your-org/nemesis/web-api:sha-abc123

Required Repository Secrets

For this workflow to function properly, ensure:

  1. Your repository has appropriate permissions to write packages
  2. GitHub Actions has permission to create and push container images

No additional secrets are needed as the workflow uses the built-in GITHUB_TOKEN.

Customization

To customize the workflow:

  • Change the registry by modifying the REGISTRY env variable
  • Update the image naming prefix in the IMAGE_PREFIX env variable
  • Add or remove services in the matrix configuration

Documentation Build and Publish Workflow

The docs.yml workflow automatically builds and publishes the Nemesis project documentation to GitHub Pages whenever documentation files are updated.

Workflow Trigger

The workflow runs on:

  • Push to the main branch when specific documentation-related files are modified:
    • Files in the docs/ directory
    • The mkdocs.yml configuration file
    • The workflow file itself (.github/workflows/docs.yml)

Environment and Permissions

The workflow runs in the github-pages environment with the following permissions:

  • pages: write: Allows publishing to GitHub Pages
  • id-token: write: Enables secure deployment
  • contents: read: Provides access to repository content

How Documentation is Generated

The workflow uses MkDocs, a popular Python-based documentation generator:

  1. Documentation source files are written in Markdown
  2. The mkdocs.yml file configures the structure and appearance
  3. MkDocs processes these files to create a static website
  4. The generated site is deployed to GitHub Pages

Accessing the Published Documentation

After a successful workflow run, the documentation is available at: https://<organization-name>.github.io/<repository-name>/

Code Vulnerability Scan Workflow

The vuln-scan.yml workflow automatically scans the Nemesis codebase for security vulnerabilities using Aqua Security's Trivy scanner.

Workflow Trigger

The workflow runs on:

  • Push to the main branch
  • Pull requests targeting the main branch (excluding changes to documentation files)