- ./dockerfiles/nemesis-submit.Dockerfile defines "specterops/nemesis-submit"
- ./scripts/monitor_folder_docker.sh pulls the config/container down and kicks everything off
- The enrichment container now tries to parse each default Yara rule and emits a single list of rules that compile correctly. Incorrect rules are now filtered out by default.
- Updated YARA rules to newest commit
- Use wget instead of git clone for pulling Yara rules down
- Changed YARA rule highlighting to Lua instead of YAML
- Fix for parsing UTF-16/UTF-32 text
- Updated/published jtr-base to include `unzip`
- Built/published jtr-base docker image based from ./dockerfiles/jtr-base.Dockerfile
- Replaced jtr compilation for encrichment and password cracker with harmj0y/jtr-base
-NLP container now splits large docs and submits them through an additional queue for embedding
-Added overall plaintext size limit that prevents larger docs from being split/indexed
-Memory and CPU limits added to NLP container to prevent starvation via the NLP container
-Average embedding vector approach eliminated due to the new streaming approach
-Additional model options pre-loaded for NLP container
-Changed default embedding model to TaylorAI/gte-tiny
-Vector and normalization added as k8s options for the NLP container
-Removed /indexing web endpoint for NLP service as text should be submitted through the Nemesis frontend
-Tokenization fixed for NLP container - now using the chosen model's tokenizer
-Removed tensorflow-serving password model as it proved not effective in production.
-Remove classic NLP summarization as it wasn't useful in production
-Changed embedding model for NLP container to one that has a proper context window
-Updated to ElasticsearchStore in NLP app
-No longer collapse all whitespace in indexing
Dashboard:
- Moved Nemesis logo to sidebar
- Edited CSS to remove blank space at top
File Viewer:
- Removed PDF rendering due to Chrome iframe sandbox blocking embeds
- All files now display as text in the Monaco viewer, irregardless of the type
- Changed Monaco to dark theme
- Add ".config" file type
Chromium page:
- Made all filters case insensitive
- Refactored into smaller functions
- Results ordered by timestamp
File Triage page:
- Refactored into smaller functions
- Changed icons for for opening as plaintext and viewing file info
- Do not hide files triaged as "unknown"
- Fixed multiple bugs in search filters not applying
- Refactored into smaller functions
- Added a link to File Viewer when uploading files
Credentials page:
- "View File Details" link now goes to File Viewer
- Fixed bug in triage not working for null values
Nosey Parker page: "View File Details" link now goes to File Viewer
Hashes page:
- Fixed bug preventing anything from being displayed
- "View Originating File" link now goes to the file viewer instead of Kibana
- Increased number of rows
- Added filtering by object_id using the URL parameters
- Added a clear filters button
Search page:
- Link to File Viewer instead of Kibana
Refactored a few of the frontend pages
Centralized auth/display templates for frontend
Fixed looping exception in Chrome history parsing (still an outstanding issue for long URLs and slow processing)
Fixed path parsing for several file parsers to not require a "\" at the beginning so they can be processed via manual upload w/o a path
Changed Chromium file parsing to account support additional Chromium profiles
Added favicon
Updated skaffold debugging settings to allow stepping into python libraries
Updated recommended VS code extensions
Add action=view_raw to the download API to render the file as text in the browser
Fixed bug in extracted text processing when submitting text with BOM to elastic
Fixed links pointing to internal API endpoints instead of public ones
Changed hash alerts to point to Hashes/File Viewer page
Allow the same credentials parsed from 2 different files to be added to the DB
Lee Christensen