mirror of
https://github.com/SpecterOps/Nemesis
synced 2026-06-08 12:36:42 +00:00
Lee Chagolla-Christensen
6ec0a3b61a
* upgrade to dapr postgresv2 statestore * actually make it v2 * dapr state table name, cleanup subscriptions/globals * proper exceptions * formatting/lint * Refactor workflow tracking and improve activity input handling - Extract workflow tracking logic into dedicated WorkflowTrackingService - Simplify activity signatures to accept specific parameters instead of generic dicts - Remove unused asyncio event loop references from enrichment modules - Update YaraRuleManager initialization and method names * re-added workflow tracking in the DB * update uvicorn prod options * enrichment work parallelism, convert queues from broadcast to task queues * Refactor pubsub and improve workflow parallelism - Split Dapr pubsub Dapr yaml components into topic-specific queues (alerting, dotnet, dpapi, files, noseyparker, workflow_monitor) - Update all Dapr volume mounts to reference new topic-specific pubsub components - Converted queues to task queues - Use YAML anchors to reduce duplication for file-enrichment replicas - Pass asyncpg pool to enrichment modules instead of creating connections - Add asyncpg_pool parameter throughout chromium and enrichment module analyzers - Update VSCode workspace (removed InspectAssembly, renamed dotnet_api to dotnet_service) - Added curl commands for Jaeger API to performance docs to help with perf troubleshooting - Created common.queues module to centralize pubsub/topic names (eases future refactoring) * worker mods * Workflow performance tuning, fix pubsub config, CLI arg changes - Fix pubsub deleteWhenUnused typo (deletedWhenUnused) - Add LOG_LEVEL environment variable support across services - CLI: Rename --repeat to --times, add --max-files option - Increase files pubsub prefetchCount from 25 to 50 - Add MAX_PARALLEL_WORKFLOWS configuration - Fix DotNetAssemblyAnalysis null handling with field validators - Update dashboard to show cumulative files/findings over time - Add RUST_LOG environment variable support to noseyparker - Update CHANGELOG for 2.1.4 release notes * Dapr 1.16.2 and use db transactions - Upgrade all Dapr containers from 1.16.1 to 1.16.2 - Reduce enrichment parallelism default from 25 to 5 workflows - Reduce healthcheck intervals from 10s to 5s for alerting and document conversion - Fixed DPAPI eventing to use new pubsubs - Refactor file_linking database operations to use atomic upserts and avoid deadlocks - Add WriteOnceViolationError handling in DPAPI masterkey analyzer - Wrap database operations in transactions for enrichment storage and plaintext indexing - Fix postgres notification handler closure variable capture * remove unused start_time * Scheduler persistence, workflow concurrency tuning, and config cleanup - Add volume for Dapr scheduler and init service - Add scheduler dependency to file enrichment service - Add async workflow client libraries - Format and cleanup compose.yaml (spacing, indentation, empty lines) * Migrate file_enrichment to async Dapr client and optimize Dockerfile - Use async DaprClient where possible in file_enrichment - Improve Dockerfile caching - Add asyncpg connection pool helper and fix typo in secret store name - Include VS Code debug configuration for document_conversion - Remove unused dapr_client from DpapiBlobAnalyzer - Clean up activity return types and better handle exceptions * Enrichment tracking for NoseyParker and logging cleanup - Add workflow_id to NoseyParkerInput and NoseyParkerOutput models - Remove workflow lookup query in noseyparker subscription handler - Adjust jaeger_perf_stats.sh output formatting and precision - Add type hints for async functions * noseyparker scanner perf, tracing for update_enrichment_results --------- Co-authored-by: Lee Chagolla-Christensen <lee@localhost>
104 lines
2.5 KiB
Docker
104 lines
2.5 KiB
Docker
# FROM nemesis-python-base-dev AS base
|
|
ARG PYTHON_BASE_DEV_IMAGE=nemesis-python-base-dev
|
|
ARG PYTHON_BASE_PROD_IMAGE=nemesis-python-base-prod
|
|
FROM ${PYTHON_BASE_DEV_IMAGE} AS base
|
|
|
|
# Install dependencies in a single RUN command to reduce layers
|
|
# Added build-essential and other build dependencies for pytsk3
|
|
RUN apt-get update && \
|
|
apt-get install -y --no-install-recommends \
|
|
curl \
|
|
libpq-dev \
|
|
libtsk-dev \
|
|
postgresql-client \
|
|
build-essential \
|
|
gcc \
|
|
g++ \
|
|
make \
|
|
autoconf \
|
|
automake \
|
|
libtool \
|
|
pkg-config \
|
|
python3-dev && \
|
|
apt-get clean && \
|
|
rm -rf /var/lib/apt/lists/*
|
|
|
|
# If dependencies change, re-copy all the dependencies
|
|
# In the future we can be more efficient with this an only copy the lib folders
|
|
# that this project uses
|
|
COPY ./projects/web_api/poetry.lock ./projects/web_api/pyproject.toml /src/projects/web_api/
|
|
|
|
COPY ./libs /src/libs
|
|
COPY ./projects/web_api /src/projects/web_api/
|
|
|
|
WORKDIR /src/projects/web_api
|
|
|
|
########################
|
|
# Development
|
|
########################
|
|
FROM base AS dev
|
|
COPY --from=base /src /src
|
|
|
|
WORKDIR /src/projects/web_api
|
|
RUN poetry install
|
|
|
|
# Immediate output (no buffering)
|
|
ENV PYTHONUNBUFFERED=1
|
|
# No .pyc/pycache files
|
|
ENV PYTHONDONTWRITEBYTECODE=1
|
|
|
|
ENV LOG_LEVEL=DEBUG
|
|
|
|
ENV UVICORN_HOST="0.0.0.0"
|
|
ENV UVICORN_PORT=8000
|
|
ENV UVICORN_WORKERS=1
|
|
ENV UVICORN_RELOAD_DIR="/src/"
|
|
|
|
ENTRYPOINT ["/bin/sh", "-c", " \
|
|
poetry run uvicorn web_api.main:app \
|
|
--host ${UVICORN_HOST} \
|
|
--port ${UVICORN_PORT} \
|
|
--workers ${UVICORN_WORKERS} \
|
|
--reload \
|
|
--reload-dir ${UVICORN_RELOAD_DIR} \
|
|
"]
|
|
|
|
########################
|
|
# Production
|
|
########################
|
|
FROM base AS bundle
|
|
COPY --from=base /src /src
|
|
|
|
WORKDIR /src/projects/web_api
|
|
RUN poetry bundle venv --python=/usr/bin/python3 --only=main /venv
|
|
|
|
FROM ${PYTHON_BASE_PROD_IMAGE} AS prod
|
|
|
|
RUN apt-get update && \
|
|
apt-get install -y \
|
|
libpq5 \
|
|
&& \
|
|
apt-get clean && \
|
|
rm -rf /var/lib/apt/lists/*
|
|
|
|
COPY --from=bundle /venv /venv
|
|
|
|
# Uvicorn production settings
|
|
ENV UVICORN_HOST=0.0.0.0 \
|
|
UVICORN_PORT=8000 \
|
|
UVICORN_PROXY_HEADERS=1 \
|
|
UVICORN_WORKERS=4 \
|
|
UVICORN_ACCESS_LOG=false \
|
|
UVICORN_LOG_LEVEL=info
|
|
|
|
# TODO: Re-enable when we're ready for release
|
|
# USER nemesis
|
|
|
|
ENTRYPOINT ["/bin/sh", "-c", "\
|
|
/venv/bin/uvicorn \"web_api.main:app\" \
|
|
--host ${UVICORN_HOST} \
|
|
--port ${UVICORN_PORT} \
|
|
--workers ${UVICORN_WORKERS} \
|
|
--proxy-headers \
|
|
--no-access-log \
|
|
"] |