mirror of
https://github.com/SpecterOps/Nemesis
synced 2026-06-08 12:36:42 +00:00
Lee Chagolla-Christensen
6ec0a3b61a
* upgrade to dapr postgresv2 statestore * actually make it v2 * dapr state table name, cleanup subscriptions/globals * proper exceptions * formatting/lint * Refactor workflow tracking and improve activity input handling - Extract workflow tracking logic into dedicated WorkflowTrackingService - Simplify activity signatures to accept specific parameters instead of generic dicts - Remove unused asyncio event loop references from enrichment modules - Update YaraRuleManager initialization and method names * re-added workflow tracking in the DB * update uvicorn prod options * enrichment work parallelism, convert queues from broadcast to task queues * Refactor pubsub and improve workflow parallelism - Split Dapr pubsub Dapr yaml components into topic-specific queues (alerting, dotnet, dpapi, files, noseyparker, workflow_monitor) - Update all Dapr volume mounts to reference new topic-specific pubsub components - Converted queues to task queues - Use YAML anchors to reduce duplication for file-enrichment replicas - Pass asyncpg pool to enrichment modules instead of creating connections - Add asyncpg_pool parameter throughout chromium and enrichment module analyzers - Update VSCode workspace (removed InspectAssembly, renamed dotnet_api to dotnet_service) - Added curl commands for Jaeger API to performance docs to help with perf troubleshooting - Created common.queues module to centralize pubsub/topic names (eases future refactoring) * worker mods * Workflow performance tuning, fix pubsub config, CLI arg changes - Fix pubsub deleteWhenUnused typo (deletedWhenUnused) - Add LOG_LEVEL environment variable support across services - CLI: Rename --repeat to --times, add --max-files option - Increase files pubsub prefetchCount from 25 to 50 - Add MAX_PARALLEL_WORKFLOWS configuration - Fix DotNetAssemblyAnalysis null handling with field validators - Update dashboard to show cumulative files/findings over time - Add RUST_LOG environment variable support to noseyparker - Update CHANGELOG for 2.1.4 release notes * Dapr 1.16.2 and use db transactions - Upgrade all Dapr containers from 1.16.1 to 1.16.2 - Reduce enrichment parallelism default from 25 to 5 workflows - Reduce healthcheck intervals from 10s to 5s for alerting and document conversion - Fixed DPAPI eventing to use new pubsubs - Refactor file_linking database operations to use atomic upserts and avoid deadlocks - Add WriteOnceViolationError handling in DPAPI masterkey analyzer - Wrap database operations in transactions for enrichment storage and plaintext indexing - Fix postgres notification handler closure variable capture * remove unused start_time * Scheduler persistence, workflow concurrency tuning, and config cleanup - Add volume for Dapr scheduler and init service - Add scheduler dependency to file enrichment service - Add async workflow client libraries - Format and cleanup compose.yaml (spacing, indentation, empty lines) * Migrate file_enrichment to async Dapr client and optimize Dockerfile - Use async DaprClient where possible in file_enrichment - Improve Dockerfile caching - Add asyncpg connection pool helper and fix typo in secret store name - Include VS Code debug configuration for document_conversion - Remove unused dapr_client from DpapiBlobAnalyzer - Clean up activity return types and better handle exceptions * Enrichment tracking for NoseyParker and logging cleanup - Add workflow_id to NoseyParkerInput and NoseyParkerOutput models - Remove workflow lookup query in noseyparker subscription handler - Adjust jaeger_perf_stats.sh output formatting and precision - Add type hints for async functions * noseyparker scanner perf, tracing for update_enrichment_results --------- Co-authored-by: Lee Chagolla-Christensen <lee@localhost>
123 lines
3.8 KiB
Docker
123 lines
3.8 KiB
Docker
# FROM nemesis-python-base-dev AS base
|
|
ARG PYTHON_BASE_DEV_IMAGE=nemesis-python-base-dev
|
|
ARG PYTHON_BASE_PROD_IMAGE=nemesis-python-base-prod
|
|
FROM ${PYTHON_BASE_DEV_IMAGE} AS base
|
|
|
|
ARG TIKA_OCR_LANGUAGES="eng"
|
|
ARG TIKA_VERSION="3.1.0"
|
|
|
|
# Install dependencies including multiple Tesseract language packs
|
|
RUN apt-get update && \
|
|
apt-get install -y --no-install-recommends \
|
|
openjdk-17-jre-headless \
|
|
wget \
|
|
tesseract-ocr \
|
|
libpq-dev \
|
|
postgresql-client \
|
|
binutils \
|
|
# Install fonts for better CJK support
|
|
fonts-noto-cjk \
|
|
fonts-noto-color-emoji && \
|
|
# Install Tesseract language packs dynamically
|
|
for lang in ${TIKA_OCR_LANGUAGES}; do \
|
|
apt-get install -y --no-install-recommends tesseract-ocr-${lang} || echo "Warning: Failed to install tesseract-ocr-${lang}"; \
|
|
done && \
|
|
# Download Tika server
|
|
wget https://archive.apache.org/dist/tika/${TIKA_VERSION}/tika-server-standard-${TIKA_VERSION}.jar -O /tika-server-standard.jar && \
|
|
apt-get clean && \
|
|
rm -rf /var/lib/apt/lists/*
|
|
|
|
# Set JAVA_HOME for JRE
|
|
ENV JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64/jre
|
|
|
|
# Copy only what's needed for dependency installation first (leveraging Docker cache)
|
|
COPY ./projects/document_conversion/poetry.lock ./projects/document_conversion/pyproject.toml /src/projects/document_conversion/
|
|
WORKDIR /src/projects/document_conversion
|
|
|
|
# Copy remaining files
|
|
COPY ./libs /src/libs
|
|
COPY ./projects/document_conversion /src/projects/document_conversion/
|
|
|
|
########################
|
|
# Development
|
|
########################
|
|
FROM base AS dev
|
|
|
|
# Set environment variables together in one layer
|
|
ENV PYTHONUNBUFFERED=1 \
|
|
PYTHONDONTWRITEBYTECODE=1 \
|
|
LOG_LEVEL=DEBUG \
|
|
UVICORN_HOST="0.0.0.0" \
|
|
UVICORN_PORT=8000 \
|
|
UVICORN_RELOAD_DIR="/src/"
|
|
|
|
RUN poetry install
|
|
|
|
ENTRYPOINT ["/bin/sh", "-c", " \
|
|
poetry run uvicorn document_conversion.main:app \
|
|
--host ${UVICORN_HOST} \
|
|
--port ${UVICORN_PORT} \
|
|
--reload \
|
|
--reload-dir ${UVICORN_RELOAD_DIR} \
|
|
"]
|
|
|
|
########################
|
|
# Runtime - Bundle the app
|
|
########################
|
|
FROM base AS bundle
|
|
|
|
RUN poetry bundle venv --python=/usr/bin/python3 --only=main /venv
|
|
|
|
########################
|
|
# Make the final image
|
|
########################
|
|
# FROM nemesis-python-base-prod AS prod
|
|
FROM ${PYTHON_BASE_PROD_IMAGE} AS prod
|
|
|
|
ARG TIKA_OCR_LANGUAGES="eng"
|
|
|
|
# Install dependencies in a single RUN command to reduce layers
|
|
RUN apt-get update && \
|
|
apt-get install -y --no-install-recommends \
|
|
openjdk-17-jre-headless \
|
|
wget \
|
|
tesseract-ocr \
|
|
libpq-dev \
|
|
postgresql-client \
|
|
binutils \
|
|
# Install fonts for better CJK support
|
|
fonts-noto-cjk \
|
|
fonts-noto-color-emoji && \
|
|
# Install Tesseract language packs dynamically
|
|
for lang in ${TIKA_OCR_LANGUAGES}; do \
|
|
apt-get install -y --no-install-recommends tesseract-ocr-${lang} || echo "Warning: Failed to install tesseract-ocr-${lang}"; \
|
|
done && \
|
|
apt-get clean && \
|
|
rm -rf /var/lib/apt/lists/*
|
|
|
|
|
|
ENV JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64/jre \
|
|
LOG_LEVEL=INFO \
|
|
UVICORN_HOST=0.0.0.0 \
|
|
UVICORN_PORT=8000 \
|
|
UVICORN_PROXY_HEADERS=1 \
|
|
UVICORN_WORKERS=1 \
|
|
UVICORN_ACCESS_LOG=false \
|
|
UVICORN_LOG_LEVEL=info
|
|
|
|
# Copy only the virtual environment from the bundle stage
|
|
COPY --from=bundle /venv /venv
|
|
# Copy the Tika JAR from base so we don't download again
|
|
COPY --from=base /tika-server-standard.jar /tika-server-standard.jar
|
|
|
|
# TODO: Re-enable when we're ready for release
|
|
# USER nemesis
|
|
|
|
ENTRYPOINT ["/bin/sh", "-c", "\
|
|
/venv/bin/uvicorn \"document_conversion.main:app\" \
|
|
--host ${UVICORN_HOST} \
|
|
--port ${UVICORN_PORT} \
|
|
--workers ${UVICORN_WORKERS} \
|
|
--proxy-headers \
|
|
--no-access-log \
|
|
"] |