admin/SpecterOps-Nemesis
1
0
Fork 0
mirror of https://github.com/SpecterOps/Nemesis synced 2026-06-08 12:36:42 +00:00
Code Issues Packages Projects Releases Wiki Activity
330 Commits 9 Branches 7 Tags
8147abb3ffbe41eb4dfcbcd00bada58e015c7477
Commit Graph

172 Commits

Author SHA1 Message Date
Max Harley a359c24caa Delete kubernetes managed state objects 2024-03-18 11:38:24 -04:00
Will 205cdc4bd4 Merge branch 'main' into helm 2024-03-08 14:20:06 -08:00
Will e8a75a5ada Fix for seatbelt json Yara rule 
- Fix for seatbelt json Yara rule
 2024-03-08 13:17:26 -08:00
Will 80b5f217a1 Update raw_date tag for submit_to_nemesis 
- Update raw_date tag for submit_to_nemesis
 2024-03-08 11:12:22 -08:00
Will 76559874b1 Added Minio storage expiration policy for bucket on creation 
- Added Minio storage expiration policy for bucket on creation
 2024-03-08 11:07:57 -08:00
Will 14ae409912 /reset route now clears minio files as well 
/reset route now clears minio files as well
 2024-03-08 00:21:12 -08:00
Will 715a7e2091 SSL endpoint revamp for docs/connectors 
- SSL endpoint revamp for docs/connectors
- Broke Jupyter's token into a separate secret
 2024-03-06 20:38:43 -08:00
Will 4874577ae3 Merge branch 'main' into helm 2024-03-06 14:29:26 -08:00
Will 031f30f469 Update NLP poetry and add Helm timeouts for skaffold 
- Updated NLP poetry packages
- Added 30min Helm timeouts for skaffold
- Set some static internal password values for compatibility
 2024-03-06 14:11:53 -08:00
Will 4927efb708 Detect and reprocess dpapi_domain_backupkey JSON 
- Detect and reprocess dpapi_domain_backupkey JSON
 2024-03-06 00:16:35 -08:00
Will 19d08a7fe7 Remove unnecessary languages for Monaco editor in File Viewer 
- Remove unnecessary languages for Monaco editor in File Viewer
 2024-03-05 11:20:31 -08:00
Will ee5ab5b330 Streamlined Yara rule parsing 
- The enrichment container now tries to parse each default Yara rule and emits a single list of rules that compile correctly. Incorrect rules are now filtered out by default.
 2024-02-27 21:45:07 -08:00
Will 23a24ef2e5 Updated YARA rules + fixes 
- Updated YARA rules to newest commit
- Use wget instead of git clone for pulling Yara rules down
- Changed YARA rule highlighting to Lua instead of YAML
- Fix for parsing UTF-16/UTF-32 text
- Updated/published jtr-base to include `unzip`
 2024-02-27 14:58:33 -08:00
Will cb8fd3066b Fix for pytorch-cpu install in NLP container 
- Fix for pytorch-cpu install in NLP container
 2024-02-26 15:50:23 -08:00
Max Harley 1228be43d7 update docs 2024-02-23 14:41:47 +01:00
Will 263c1e9309 Text processing mods 
- Added basic text cleaning
- Upped Tika version
- Added memory/cpu limits to Tika
- Don't use Tika for plaintext files
- Upped timeout for Tika extraction to 120 seconds
 2024-02-15 21:06:10 -08:00
Will 7f48dff8e9 Fix for searching over large files in Elasticsearch 
- Fix for searching over large files in Elasticsearch
 2024-02-13 16:47:22 -08:00
Will 825094cbee Fix for multiple filters for semantic search 
- Fix for multiple filters for semantic search
 2024-02-13 15:15:02 -08:00
Will ad45a74cda Tweaks 
- Newest files now at the top in the Files view of Dashboard
- Adjusted CPU limits for NLP/elastic
- Enabled fuzziness for NLP elastic search
 2024-02-09 17:14:57 -08:00
Will 66c2489893 Merge pull request #42 from SpecterOps/nlp-improved-search 
NLP improved search
 2024-02-09 16:52:33 -08:00
Will 8a702219b7 Renamed "Text Chunk Search" to "Snippet Search" in the dashboard 
-Renamed "Text Chunk Search" to "Snippet Search" in the dashboard for clarity
 2024-02-09 16:49:36 -08:00
Will e2fb068002 Multiple NLP mods 
-Can now have multiple include/exclude path filters, separated by |
-Collapse whitespace in chunk display
-Fixed candidates for fuzzy/semantic search
-Now weight RRF towards fuzzy search
-Min/max scale RRF scores based on top BM25 score
-For hybrid search, a net of 3*num_results is now cast for semantic/fuzzy searches, condensed down by reciprocal rank fusion
 2024-02-09 16:35:09 -08:00
Will a6a95ca5cc Update requirements.txt for Mythic Connector 
Bumped aiohttp down do it works with the mythic pypi package
 2024-02-09 13:03:33 -08:00
Will 3f1bf4e4f5 Expanded explanation "About Document Search" expander 
-Expanded explanation "About Document Search" expander
 2024-02-08 18:13:41 -08:00
Will af3f532d51 Combined Source Code search page into Document Search 
-Combined Source Code search page into Document Search with an index selection option
 2024-02-08 18:09:54 -08:00
Will 465a0f02f6 Redid semantic search to be RRF hybrid search 
-Redid semantic search to be RRF hybrid search
-New view in the document search dashboard for text chunks
 2024-02-08 17:49:15 -08:00
Will 3d458bdd85 POSSIBLY BREAKING CHANGE - Bumped Elasticsearch to 8.11.0 
-POSSIBLY BREAKING CHANGE - Bumped Elasticsearch to 8.11.0
-Upgraded appropriate client packages
 2024-02-08 17:42:16 -08:00
Will 5cd33059c7 Added path metadata filtering for all document searches 
-Added path metadata filtering for all document searches
 2024-02-07 17:36:54 -08:00
Will fedf4a06fc Added plaintext limit to cracklist updates 
-Added plaintext limit to cracklist updates
 2024-02-07 14:06:14 -08:00
Will d3f1516f90 Fixed plaintext size restriction for Elastic ingest in dev 
-Fixed plaintext size restriction for Elastic ingest in dev
 2024-02-07 13:55:47 -08:00
Will 5b24179120 NLP modifications 
-NLP container now splits large docs and submits them through an additional queue for embedding
-Added overall plaintext size limit that prevents larger docs from being split/indexed
-Memory and CPU limits added to NLP container to prevent starvation via the NLP container
-Average embedding vector approach eliminated due to the new streaming approach
-Additional model options pre-loaded for NLP container
-Changed default embedding model to TaylorAI/gte-tiny
-Vector and normalization added as k8s options for the NLP container
-Removed /indexing web endpoint for NLP service as text should be submitted through the Nemesis frontend
-Tokenization fixed for NLP container - now using the chosen model's tokenizer
 2024-02-07 00:59:39 -08:00
Will 3b14008fb6 Bumped fastapi version for 4 affected containers due to dependabot issues 
-Bumped fastapi version for 4 affected containers due to dependabot issues
 2024-02-05 12:53:09 -08:00
Lee Christensen d9d787e2ed return raw HTLM on landing page 2024-02-02 16:58:20 -05:00
Will 052e6960a1 Redid Tika detection 
-Redid Tika detection to use mime types instead of extensions
-Fixes for document search display
 2024-02-01 15:51:08 -08:00
Will fef7af0a96 Added weights to the calculation of the average document embedding vector 
-Added weights to the calculation of the average document embedding vector
 2024-02-01 14:58:33 -08:00
Will 73d1771205 Restricted two input queues to 1 doc at a time 
-Restricted input queue for NLP container to 1 doc at a time
-Restricted input queue for file ingest in enrichment to 1 doc at a time
 2024-02-01 14:07:47 -08:00
Will 3f81957c33 Added document average embedding vector indexing 
-Added document average embedding vector indexing
 2024-02-01 14:06:16 -08:00
Will dabbe94b8b Merge pull request #39 from SpecterOps/tensorflow-removal 
Removed tensorflow-serving password model
 2024-02-01 12:11:34 -08:00
Will 416f7642ed Removed tensorflow-serving password model 
-Removed tensorflow-serving password model as it proved not effective in production.
-Remove classic NLP summarization as it wasn't useful in production
 2024-02-01 12:11:12 -08:00
Max Harley 17970d4b5a unbundle from extension-cli (#38) 2024-02-01 12:52:14 -05:00
Will 25743a5ae5 Batching for text vectorization in nlp container 
-Batching for text vectorization in `nlp` container
 2024-01-31 22:32:40 -08:00
Will ecc6ab8287 Bump for urllib3 for dashboard 
-Bump for urllib3 for dashboard
 2024-01-31 11:28:50 -08:00
Will b2c0d61a29 Bump for requests in stage1 connector 
-Bump for requests in stage1 connector
 2024-01-31 11:27:28 -08:00
Will d2f007ac1f Bump requests library for mythic connector 
Bump requests library for mythic connector
 2024-01-31 11:24:04 -08:00
Will 6a3d8faa9b Bump aiohttp for mythic and sliver connectors 
-Bump aiohttp for mythic and sliver connectors
 2024-01-31 11:22:47 -08:00
Will 955d7935b0 fastapi/aiohttp bump and remove dependency for fastapi-class for enrichment 
-fastapi/aiohttp bump and remove dependency for fastapi-class for `enrichment`
 2024-01-31 11:11:48 -08:00
Will cd7655e1db Bump fastapi and aiohttp for passwordcracker 
-Bump fastapi and aiohttp for `passwordcracker`
 2024-01-30 18:02:07 -08:00
Will 419a4ca6f7 fastapi bump and remove dependency for fastapi-class for nlp 
-fastapi bump and remove dependency for fastapi-class for `nlp`
-bump aiohttp
 2024-01-30 18:01:45 -08:00
Will 78a1554632 fastapi bump and remove dependency for fastapi-class 
-fastapi bump and remove dependency for fastapi-class
 2024-01-30 16:47:09 -08:00
Will 186b494e55 Bump aiohttp package in dotnet container 
-Bump aiohttp package in `dotnet` container
 2024-01-30 15:29:45 -08:00