mirror of
https://github.com/SpecterOps/Nemesis
synced 2026-06-08 12:36:42 +00:00
Lee Chagolla-Christensen
b63813e300
- Add centralized PostgreSQL connection string helper in common/db.py - Replace ntpath with posixpath - Add get_drive_from_path() helper for extracting drive letters from paths - Update all modules to use centralized DB connection helper instead of individual Dapr client calls - Improve path normalization across Chromium, DPAPI, and file enrichment modules - Added dependencies to local libraries - Expand file linking tests with comprehensive rule engine test cases - Refactor masterkey path construction to use posixpath.normpath for proper path joining - Update state helpers and API models to use centralized connection - Updated web API to use strongly typed models instead of dicts internally - Improve DPAPI core functionality with better error handling and path normalization
Document Conversion Service
A microservice for the Nemesis platform that handles document processing, text extraction, and file format conversion.
Purpose
This service processes uploaded files to extract textual content, convert documents to PDF format, and extract strings from binary files. It serves as a key component in making file contents searchable and viewable within the Nemesis platform.
Features
- Text extraction: Extract readable text from various document formats using Apache Tika
- PDF conversion: Convert Office documents and other formats to PDF using Gotenberg
- String extraction: Extract printable strings from binary files
- Encryption detection: Automatically detect and skip encrypted or protected files
- Parallel processing: Execute multiple extraction methods simultaneously using Dapr workflows
- Container support: Process files extracted from archives and containers
Document Processing Pipeline
The service uses a Dapr workflow to process files through three parallel activities:
- Tika text extraction: Extracts formatted text from documents
- String extraction: Pulls ASCII/Unicode strings from binary files
- PDF conversion: Converts supported formats to PDF for viewing
Supported File Types
- Text extraction: Office documents, PDFs, HTML, RTF, and other text-based formats
- PDF conversion: Microsoft Office files, LibreOffice documents, text files
- String extraction: Any binary file format
Configuration
The service integrates with external components:
- Apache Tika: Java-based text extraction engine
- Gotenberg: Document to PDF conversion service via HTTP API
- Minio: Object storage for input files and generated outputs
- PostgreSQL: Workflow state and transform metadata storage
Workflow Behavior
Files are processed only if they meet specific criteria:
- Not already plaintext
- Not encrypted or password protected
- Either original submissions or container-extracted files
- Not existing transforms to avoid duplicate processing
Health Monitoring
GET /healthz: Comprehensive health check including database connectivity, JVM status, and workflow runtime verification