6cebd39a2b
Auto-build text_embeddings Elastic index for predictable index ID
...
- Auto-build `text_embeddings` Elastic index for predictable index ID
2024-03-06 00:16:03 -08:00
19d08a7fe7
Remove unnecessary languages for Monaco editor in File Viewer
...
- Remove unnecessary languages for Monaco editor in File Viewer
2024-03-05 11:20:31 -08:00
93e0a5a64c
Mark base Yara rules to a specific commit
...
- Mark base Yara rules to a specific commit
2024-03-05 10:48:18 -08:00
ee5ab5b330
Streamlined Yara rule parsing
...
- The enrichment container now tries to parse each default Yara rule and emits a single list of rules that compile correctly. Incorrect rules are now filtered out by default.
2024-02-27 21:45:07 -08:00
23a24ef2e5
Updated YARA rules + fixes
...
- Updated YARA rules to newest commit
- Use wget instead of git clone for pulling Yara rules down
- Changed YARA rule highlighting to Lua instead of YAML
- Fix for parsing UTF-16/UTF-32 text
- Updated/published jtr-base to include `unzip`
2024-02-27 14:58:33 -08:00
d5714c4ddb
jtr-base Docker image
...
- Built/published jtr-base docker image based from ./dockerfiles/jtr-base.Dockerfile
- Replaced jtr compilation for encrichment and password cracker with harmj0y/jtr-base
2024-02-27 13:18:31 -08:00
cb8fd3066b
Fix for pytorch-cpu install in NLP container
...
- Fix for pytorch-cpu install in NLP container
2024-02-26 15:50:23 -08:00
13a11129df
unbundle from extension-cli ( #45 )
2024-02-23 08:35:04 +01:00
7819ef6981
Added example named_pipe structured json
...
- Added example named_pipe structured json
2024-02-16 16:37:28 -08:00
bde89e1623
Merge pull request #44 from SpecterOps/docs-improvement
...
Docs improvement
2024-02-15 21:08:59 -08:00
862a3f4a61
Merge pull request #43 from SpecterOps/main
...
catch up to main
2024-02-15 21:08:17 -08:00
263c1e9309
Text processing mods
...
- Added basic text cleaning
- Upped Tika version
- Added memory/cpu limits to Tika
- Don't use Tika for plaintext files
- Upped timeout for Tika extraction to 120 seconds
2024-02-15 21:06:10 -08:00
acb437ad01
Added documentation
...
- Added Usage Guide
- Added Hasura endpoint documentation
- Misc fixups
2024-02-13 17:56:54 -08:00
7f48dff8e9
Fix for searching over large files in Elasticsearch
...
- Fix for searching over large files in Elasticsearch
2024-02-13 16:47:22 -08:00
825094cbee
Fix for multiple filters for semantic search
...
- Fix for multiple filters for semantic search
2024-02-13 15:15:02 -08:00
edc09187cc
Bumped NLP memory limit
...
- Bumped NLP memory limit to 1500 from 1000
2024-02-13 11:55:07 -08:00
cf6f90ac29
Added Nemesis route for nlp service
...
- Added Nemesis route (/nlp/) for `nlp` service
2024-02-12 17:26:08 -08:00
ad45a74cda
Tweaks
...
- Newest files now at the top in the Files view of Dashboard
- Adjusted CPU limits for NLP/elastic
- Enabled fuzziness for NLP elastic search
2024-02-09 17:14:57 -08:00
66c2489893
Merge pull request #42 from SpecterOps/nlp-improved-search
...
NLP improved search
2024-02-09 16:52:33 -08:00
8a702219b7
Renamed "Text Chunk Search" to "Snippet Search" in the dashboard
...
-Renamed "Text Chunk Search" to "Snippet Search" in the dashboard for clarity
2024-02-09 16:49:36 -08:00
e2fb068002
Multiple NLP mods
...
-Can now have multiple include/exclude path filters, separated by |
-Collapse whitespace in chunk display
-Fixed candidates for fuzzy/semantic search
-Now weight RRF towards fuzzy search
-Min/max scale RRF scores based on top BM25 score
-For hybrid search, a net of 3*num_results is now cast for semantic/fuzzy searches, condensed down by reciprocal rank fusion
2024-02-09 16:35:09 -08:00
a6a95ca5cc
Update requirements.txt for Mythic Connector
...
Bumped aiohttp down do it works with the mythic pypi package
2024-02-09 13:03:33 -08:00
3f1bf4e4f5
Expanded explanation "About Document Search" expander
...
-Expanded explanation "About Document Search" expander
2024-02-08 18:13:41 -08:00
af3f532d51
Combined Source Code search page into Document Search
...
-Combined Source Code search page into Document Search with an index selection option
2024-02-08 18:09:54 -08:00
465a0f02f6
Redid semantic search to be RRF hybrid search
...
-Redid semantic search to be RRF hybrid search
-New view in the document search dashboard for text chunks
2024-02-08 17:49:15 -08:00
3d458bdd85
POSSIBLY BREAKING CHANGE - Bumped Elasticsearch to 8.11.0
...
-POSSIBLY BREAKING CHANGE - Bumped Elasticsearch to 8.11.0
-Upgraded appropriate client packages
2024-02-08 17:42:16 -08:00
1297d01be2
Merge pull request #41 from SpecterOps/nlp-throughput-mods
...
NLP throughput mods
2024-02-07 17:51:47 -08:00
5cd33059c7
Added path metadata filtering for all document searches
...
-Added path metadata filtering for all document searches
2024-02-07 17:36:54 -08:00
fedf4a06fc
Added plaintext limit to cracklist updates
...
-Added plaintext limit to cracklist updates
2024-02-07 14:06:14 -08:00
134a561598
Changed plaintext indexing limit to 50 megs
...
Changed plaintext indexing limit to 50 megs
2024-02-07 13:57:55 -08:00
d3f1516f90
Fixed plaintext size restriction for Elastic ingest in dev
...
-Fixed plaintext size restriction for Elastic ingest in dev
2024-02-07 13:55:47 -08:00
5b24179120
NLP modifications
...
-NLP container now splits large docs and submits them through an additional queue for embedding
-Added overall plaintext size limit that prevents larger docs from being split/indexed
-Memory and CPU limits added to NLP container to prevent starvation via the NLP container
-Average embedding vector approach eliminated due to the new streaming approach
-Additional model options pre-loaded for NLP container
-Changed default embedding model to TaylorAI/gte-tiny
-Vector and normalization added as k8s options for the NLP container
-Removed /indexing web endpoint for NLP service as text should be submitted through the Nemesis frontend
-Tokenization fixed for NLP container - now using the chosen model's tokenizer
2024-02-07 00:59:39 -08:00
3b14008fb6
Bumped fastapi version for 4 affected containers due to dependabot issues
...
-Bumped fastapi version for 4 affected containers due to dependabot issues
2024-02-05 12:53:09 -08:00
d9d787e2ed
return raw HTLM on landing page
2024-02-02 16:58:20 -05:00
052e6960a1
Redid Tika detection
...
-Redid Tika detection to use mime types instead of extensions
-Fixes for document search display
2024-02-01 15:51:08 -08:00
fef7af0a96
Added weights to the calculation of the average document embedding vector
...
-Added weights to the calculation of the average document embedding vector
2024-02-01 14:58:33 -08:00
b41093d123
Merge pull request #40 from SpecterOps/nlp_mods
...
Nlp mods
2024-02-01 14:08:04 -08:00
73d1771205
Restricted two input queues to 1 doc at a time
...
-Restricted input queue for NLP container to 1 doc at a time
-Restricted input queue for file ingest in enrichment to 1 doc at a time
2024-02-01 14:07:47 -08:00
3f81957c33
Added document average embedding vector indexing
...
-Added document average embedding vector indexing
2024-02-01 14:06:16 -08:00
dabbe94b8b
Merge pull request #39 from SpecterOps/tensorflow-removal
...
Removed tensorflow-serving password model
2024-02-01 12:11:34 -08:00
416f7642ed
Removed tensorflow-serving password model
...
-Removed tensorflow-serving password model as it proved not effective in production.
-Remove classic NLP summarization as it wasn't useful in production
2024-02-01 12:11:12 -08:00
17970d4b5a
unbundle from extension-cli ( #38 )
2024-02-01 12:52:14 -05:00
25743a5ae5
Batching for text vectorization in nlp container
...
-Batching for text vectorization in `nlp` container
2024-01-31 22:32:40 -08:00
ecc6ab8287
Bump for urllib3 for dashboard
...
-Bump for urllib3 for dashboard
2024-01-31 11:28:50 -08:00
b2c0d61a29
Bump for requests in stage1 connector
...
-Bump for requests in stage1 connector
2024-01-31 11:27:28 -08:00
d2f007ac1f
Bump requests library for mythic connector
...
Bump requests library for mythic connector
2024-01-31 11:24:04 -08:00
6a3d8faa9b
Bump aiohttp for mythic and sliver connectors
...
-Bump aiohttp for mythic and sliver connectors
2024-01-31 11:22:47 -08:00
05c088f485
Merge pull request #37 from SpecterOps/dependabot-fixes
...
Dependabot fixes
2024-01-31 11:19:33 -08:00
cc0ebb89e5
Bumped cryptography package in nemesiscommon
...
-Bumped cryptography package in nemesiscommon
2024-01-31 11:18:35 -08:00
098e6c496f
Bumped pycryptodomex package in nemesiscommon
...
-Bumped pycryptodomex package in nemesiscommon
2024-01-31 11:17:01 -08:00
Will