17b664dc0d
Merge remote-tracking branch 'origin/main' into host_analysis
2024-05-01 01:07:50 -04:00
f42f879f31
add dockerignore, trim docker image sizes
2024-04-24 11:35:17 -04:00
743b6f715f
speed up nlp/enrichment builds, decrease image size via dockerignore
2024-04-24 04:17:17 -04:00
750cb69cb6
update lock files, protobuf update
2024-04-23 03:37:55 -04:00
298ed4ba00
Merge branch 'main' into host_analysis
2024-04-23 03:24:46 -04:00
ba0f3ce6c0
Lots of changes...
...
more test examples, readd registry/service objects
fix agentmapping, process, host
fix env settings
add DB files
update tables, ruff, formatting, lots of changes...
Merge remote-tracking branch 'origin/main' into host_analysis
return raw HTLM on landing page
and more...
2024-04-23 02:14:21 -04:00
8a86cbf9e9
Update references to operation-creds in scripts and documentation
2024-04-22 20:15:04 -04:00
10ba778641
Increased timeout for submit_to_nemesis.py
...
- Increased timeout for `submit_to_nemesis.py` to 15 min
- Modified `nemesis-submit.Dockerfile` to work for macOS/arm builds
2024-04-08 13:13:39 -07:00
679f1e3a76
Fixes for large file processing
...
- Fixes for large file processing
2024-04-05 15:52:25 -07:00
e4cc815a68
Update helpers.py
2024-04-05 11:46:08 -07:00
a62c0e9857
Start of large file handling
...
- Start of large file handling
2024-04-05 10:08:28 -07:00
4489ff2dab
Get NoseyParker rules from folder
2024-04-02 16:39:48 -04:00
0f44413518
Moved two env vars to values.yaml
...
- Moved STREAMLIT_SERVER_MAX_UPLOAD_SIZE in dashboard to values.yaml
- Moved EXTRACTED_ARCHIVE_SIZE_LIMIT in enrichment to values.yaml
2024-03-27 15:30:41 -07:00
9a0c00e354
Fix for self-signed certs for submit_to_nemesis
...
- Fix for self-signed certs for submit_to_nemesis
2024-03-26 15:17:51 -07:00
47ea5100c6
Added JAR processing
...
- Added JAR processing
2024-03-19 23:22:16 -07:00
a359c24caa
Delete kubernetes managed state objects
2024-03-18 11:38:24 -04:00
205cdc4bd4
Merge branch 'main' into helm
2024-03-08 14:20:06 -08:00
e8a75a5ada
Fix for seatbelt json Yara rule
...
- Fix for seatbelt json Yara rule
2024-03-08 13:17:26 -08:00
80b5f217a1
Update raw_date tag for submit_to_nemesis
...
- Update raw_date tag for submit_to_nemesis
2024-03-08 11:12:22 -08:00
76559874b1
Added Minio storage expiration policy for bucket on creation
...
- Added Minio storage expiration policy for bucket on creation
2024-03-08 11:07:57 -08:00
14ae409912
/reset route now clears minio files as well
...
/reset route now clears minio files as well
2024-03-08 00:21:12 -08:00
715a7e2091
SSL endpoint revamp for docs/connectors
...
- SSL endpoint revamp for docs/connectors
- Broke Jupyter's token into a separate secret
2024-03-06 20:38:43 -08:00
4874577ae3
Merge branch 'main' into helm
2024-03-06 14:29:26 -08:00
031f30f469
Update NLP poetry and add Helm timeouts for skaffold
...
- Updated NLP poetry packages
- Added 30min Helm timeouts for skaffold
- Set some static internal password values for compatibility
2024-03-06 14:11:53 -08:00
4927efb708
Detect and reprocess dpapi_domain_backupkey JSON
...
- Detect and reprocess dpapi_domain_backupkey JSON
2024-03-06 00:16:35 -08:00
19d08a7fe7
Remove unnecessary languages for Monaco editor in File Viewer
...
- Remove unnecessary languages for Monaco editor in File Viewer
2024-03-05 11:20:31 -08:00
ee5ab5b330
Streamlined Yara rule parsing
...
- The enrichment container now tries to parse each default Yara rule and emits a single list of rules that compile correctly. Incorrect rules are now filtered out by default.
2024-02-27 21:45:07 -08:00
23a24ef2e5
Updated YARA rules + fixes
...
- Updated YARA rules to newest commit
- Use wget instead of git clone for pulling Yara rules down
- Changed YARA rule highlighting to Lua instead of YAML
- Fix for parsing UTF-16/UTF-32 text
- Updated/published jtr-base to include `unzip`
2024-02-27 14:58:33 -08:00
cb8fd3066b
Fix for pytorch-cpu install in NLP container
...
- Fix for pytorch-cpu install in NLP container
2024-02-26 15:50:23 -08:00
1228be43d7
update docs
2024-02-23 14:41:47 +01:00
263c1e9309
Text processing mods
...
- Added basic text cleaning
- Upped Tika version
- Added memory/cpu limits to Tika
- Don't use Tika for plaintext files
- Upped timeout for Tika extraction to 120 seconds
2024-02-15 21:06:10 -08:00
7f48dff8e9
Fix for searching over large files in Elasticsearch
...
- Fix for searching over large files in Elasticsearch
2024-02-13 16:47:22 -08:00
825094cbee
Fix for multiple filters for semantic search
...
- Fix for multiple filters for semantic search
2024-02-13 15:15:02 -08:00
ad45a74cda
Tweaks
...
- Newest files now at the top in the Files view of Dashboard
- Adjusted CPU limits for NLP/elastic
- Enabled fuzziness for NLP elastic search
2024-02-09 17:14:57 -08:00
66c2489893
Merge pull request #42 from SpecterOps/nlp-improved-search
...
NLP improved search
2024-02-09 16:52:33 -08:00
8a702219b7
Renamed "Text Chunk Search" to "Snippet Search" in the dashboard
...
-Renamed "Text Chunk Search" to "Snippet Search" in the dashboard for clarity
2024-02-09 16:49:36 -08:00
e2fb068002
Multiple NLP mods
...
-Can now have multiple include/exclude path filters, separated by |
-Collapse whitespace in chunk display
-Fixed candidates for fuzzy/semantic search
-Now weight RRF towards fuzzy search
-Min/max scale RRF scores based on top BM25 score
-For hybrid search, a net of 3*num_results is now cast for semantic/fuzzy searches, condensed down by reciprocal rank fusion
2024-02-09 16:35:09 -08:00
a6a95ca5cc
Update requirements.txt for Mythic Connector
...
Bumped aiohttp down do it works with the mythic pypi package
2024-02-09 13:03:33 -08:00
3f1bf4e4f5
Expanded explanation "About Document Search" expander
...
-Expanded explanation "About Document Search" expander
2024-02-08 18:13:41 -08:00
af3f532d51
Combined Source Code search page into Document Search
...
-Combined Source Code search page into Document Search with an index selection option
2024-02-08 18:09:54 -08:00
465a0f02f6
Redid semantic search to be RRF hybrid search
...
-Redid semantic search to be RRF hybrid search
-New view in the document search dashboard for text chunks
2024-02-08 17:49:15 -08:00
3d458bdd85
POSSIBLY BREAKING CHANGE - Bumped Elasticsearch to 8.11.0
...
-POSSIBLY BREAKING CHANGE - Bumped Elasticsearch to 8.11.0
-Upgraded appropriate client packages
2024-02-08 17:42:16 -08:00
5cd33059c7
Added path metadata filtering for all document searches
...
-Added path metadata filtering for all document searches
2024-02-07 17:36:54 -08:00
fedf4a06fc
Added plaintext limit to cracklist updates
...
-Added plaintext limit to cracklist updates
2024-02-07 14:06:14 -08:00
d3f1516f90
Fixed plaintext size restriction for Elastic ingest in dev
...
-Fixed plaintext size restriction for Elastic ingest in dev
2024-02-07 13:55:47 -08:00
5b24179120
NLP modifications
...
-NLP container now splits large docs and submits them through an additional queue for embedding
-Added overall plaintext size limit that prevents larger docs from being split/indexed
-Memory and CPU limits added to NLP container to prevent starvation via the NLP container
-Average embedding vector approach eliminated due to the new streaming approach
-Additional model options pre-loaded for NLP container
-Changed default embedding model to TaylorAI/gte-tiny
-Vector and normalization added as k8s options for the NLP container
-Removed /indexing web endpoint for NLP service as text should be submitted through the Nemesis frontend
-Tokenization fixed for NLP container - now using the chosen model's tokenizer
2024-02-07 00:59:39 -08:00
3b14008fb6
Bumped fastapi version for 4 affected containers due to dependabot issues
...
-Bumped fastapi version for 4 affected containers due to dependabot issues
2024-02-05 12:53:09 -08:00
d9d787e2ed
return raw HTLM on landing page
2024-02-02 16:58:20 -05:00
4761e97f16
Merge remote-tracking branch 'origin/main' into host_analysis
2024-02-01 19:21:59 -05:00
052e6960a1
Redid Tika detection
...
-Redid Tika detection to use mime types instead of extensions
-Fixes for document search display
2024-02-01 15:51:08 -08:00
Lee Christensen