admin/splunk-security_content
1
0
Fork 0
mirror of https://github.com/splunk/security_content synced 2026-06-08 17:32:49 +00:00
Code Issues Packages Projects Releases Wiki Activity

Branch was auto-updated.

Browse Source
This commit is contained in:
srv-rr-gh-researchbt
2023-04-12 11:21:26 -07:00
committed by GitHub
parent 7ad51e48b6 000a7bb6a7
commit 6710e09236
3262 changed files with 132011 additions and 105138 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/code-testing.yml
-36
View File
@@ -1,36 +0,0 @@
name: code-testing
on:
push:
jobs:
code-testing:
runs-on: ubuntu-latest
steps:
- name: Check out the repository code
uses: actions/checkout@v2
- uses: actions/setup-python@v2
with:
python-version: '3.9' #Available versions here - https://github.com/actions/python-versions/releases easy to change/make a matrix/use pypy
architecture: 'x64' # optional x64 or x86. Defaults to x64 if not specified
- name: Install System Packages
run: |
sudo apt update -qq
sudo apt install jq -qq
- name: Install Python Dependencies
run: |
#Get the virtualenv set up
rm -rf venv
python -m venv --clear venv
source venv/bin/activate
python -m pip install --upgrade pip
python -m pip install wheel
python -m pip install -q -r requirements.txt
- name: test code with pytest
run: |
source venv/bin/activate
export PYTHONPATH=$PYTHONPATH:/home/runner/work/security_content
pytest -s bin/contentctl_project
.github/workflows/semgrep-analysis.yml
-58
View File
@@ -1,58 +0,0 @@
# This workflow file requires a free account on Semgrep.dev to
# manage rules, file ignores, notifications, and more.
#
# See https://semgrep.dev/docs
name: Semgrep
on:
push:
pull_request:
types: [opened, reopened]
jobs:
validate-tag-if-present:
runs-on: ubuntu-latest
steps:
- name: TAGGED, Validate that the tag is in the correct format
run: |
echo "The GITHUB_REF: $GITHUB_REF"
#First check to see if the release is a tag
if [[ $GITHUB_REF =~ refs/tags/* ]]; then
#Yes, this is a tag, so we need to test to make sure that the tag
#is in the correct format (like v1.10.20)
if [[ $GITHUB_REF =~ refs/tags/v[0-9]+.[0-9]+.[0-9]+ ]]; then
echo "PASS: Tagged release with good format"
exit 0
else
echo "FAIL: Tagged release with bad format"
exit 1
fi
else
echo "PASS: Not a tagged release"
exit 0
fi
semgrep:
name: Scan
runs-on: ubuntu-latest
steps:
# Checkout project source
- uses: actions/checkout@v2
# Scan code using project's configuration on https://semgrep.dev/manage
- uses: returntocorp/semgrep-action@v1
with:
generateSarif: "1"
config: >- # more at semgrep.dev/explore
p/security-audit
p/secrets
# Upload SARIF file generated in previous step
#The following lines are commented out right now pending a fix to the semgrep repo
- name: Upload SARIF file
uses: github/codeql-action/upload-sarif@v1
with:
sarif_file: semgrep.sarif
if: always()
README.md
+11
View File
@@ -53,6 +53,7 @@ The Content Control tool allows you to manipulate Splunk Security Content via th
4. **build** - Builds an application suitable for deployment on a search head using Slim, the Splunk Packaging Toolkit
5. **inspect** - Uses a local version of appinspect to ensure that the app you built meets basic quality standards.
6. **cloud_deploy** - Using ACS, deploy your custom app to a running Splunk Cloud Instance.
7. **convert** - Convert a detection rule with sigma syntax to a Splunk SPL detection
### pre-requisites
Make sure you use python version 3.9.
@@ -79,6 +80,16 @@ for a more indepth write up on how to write content see our [guide](https://gith
### generate a splunk app from current content
`python contentctl.py -p . generate -o dist/escu -pr ESCU`
### convert a Sigma search into a Splunk detection
Detection rule using tstats and cim datamodel:
`python contentctl.py -p . convert -dm cim -o detections/endpoint/ -dp dev/endpoint/attempted_credential_dump_from_registry_via_reg_exe.yml`
Detection rule using raw:
`python contentctl.py -p . convert -dm raw -o detections/endpoint/ -dp dev/endpoint/attempted_credential_dump_from_registry_via_reg_exe.yml`
Detection rule converted to Windows Security Event Code 4688:
`python contentctl.py -p . convert -dm raw -lo "Windows Security 4688" -o detections/endpoint/ -dp dev/endpoint/attempted_credential_dump_from_registry_via_reg_exe.yml`
# MITRE ATT&CK ⚔️
### Detection Coverage
To view an up-to-date detection coverage map for all the content tagged with MITRE techniques visit: [https://mitremap.splunkresearch.com/](https://mitremap.splunkresearch.com/) under the **Detection Coverage** layer. Below is a snapshot in time of what technique we currently have some detection coverage for. The darker the shade of blue the more detections we have for this particular technique. This map is automatically updated on every release and generated from the [generate-coverage-map.py](https://github.com/splunk/security_content/blob/develop/bin/generate-coverage-map.py).
baselines/baseline_of_blocked_outbound_traffic_from_aws.yml
+1 -3
View File
@@ -27,9 +27,7 @@ tags:
analytic_story:
- AWS Network ACL Activity
- Suspicious AWS Traffic
- Command and Control
deployments:
- Daily Cache Updates
- Command And Control
detections:
- Detect Spike in blocked Outbound Traffic from your AWS
product:
baselines/baseline_of_cloud_infrastructure_api_calls_per_user.yml
+6 -2
View File
@@ -34,8 +34,6 @@ references: []
tags:
analytic_story:
- Suspicious Cloud User Activities
deployments:
- Weekly Model Rebuild 90 Day Lookback
detections:
- Abnormally High Number Of Cloud Infrastructure API Calls
product:
@@ -47,3 +45,9 @@ tags:
- All_Changes.user
- All_Changes.status
security_domain: network
deployment:
scheduling:
cron_schedule: 0 2 * * 0
earliest_time: -90d@d
latest_time: -1d@d
schedule_window: auto
baselines/baseline_of_cloud_instances_destroyed.yml
+6 -2
View File
@@ -37,8 +37,6 @@ tags:
analytic_story:
- Suspicious Cloud Instance Activities
- Cloud Cryptomining
deployments:
- Weekly Model Rebuild 90 Day Lookback
detections:
- Abnormally High Number Of Cloud Instances Destroyed
product:
@@ -51,3 +49,9 @@ tags:
- All_Changes.status
- All_Changes.object_category
security_domain: network
deployment:
scheduling:
cron_schedule: 0 2 * * 0
earliest_time: -90d@d
latest_time: -1d@d
schedule_window: auto
baselines/baseline_of_cloud_instances_launched.yml
+6 -2
View File
@@ -37,8 +37,6 @@ tags:
analytic_story:
- Cloud Cryptomining
- Suspicious Cloud Instance Activities
deployments:
- Weekly Model Rebuild 90 Day Lookback
detections:
- Abnormally High Number Of Cloud Instances Launched
product:
@@ -51,3 +49,9 @@ tags:
- All_Changes.status
- All_Changes.object_category
security_domain: network
deployment:
scheduling:
cron_schedule: 0 2 * * 0
earliest_time: -90d@d
latest_time: -1d@d
schedule_window: auto
baselines/baseline_of_cloud_security_group_api_calls_per_user.yml
+6 -2
View File
@@ -33,8 +33,6 @@ references: []
tags:
analytic_story:
- Suspicious Cloud User Activities
deployments:
- Weekly Model Rebuild 90 Day Lookback
detections:
- Abnormally High Number Of Cloud Security Group API Calls
product:
@@ -47,3 +45,9 @@ tags:
- All_Changes.status
- All_Changes.object_category
security_domain: network
deployment:
scheduling:
cron_schedule: 0 2 * * 0
earliest_time: -90d@d
latest_time: -1d@d
schedule_window: auto
baselines/baseline_of_command_line_length___mltk.yml
+1 -2
View File
@@ -34,8 +34,6 @@ tags:
- Suspicious Command-Line Executions
- Suspicious MSHTA Activity
- Unusual Processes
deployments:
- Daily Cache Updates
detections:
- Detect Prohibited Applications Spawning cmd.exe
- Unusually Long Command Line - MLTK
@@ -50,3 +48,4 @@ tags:
- Processes.process_name
- Processes.process
security_domain: endpoint
baselines/baseline_of_dns_query_length___mltk.yml
+1 -3
View File
@@ -29,9 +29,7 @@ tags:
analytic_story:
- Hidden Cobra Malware
- Suspicious DNS Traffic
- Command and Control
deployments:
- Daily Cache Updates
- Command And Control
detections:
- DNS Query Length Outliers - MLTK
product:
baselines/baseline_of_network_acl_activity_by_arn.yml
-2
View File
@@ -23,8 +23,6 @@ references: []
tags:
analytic_story:
- AWS Network ACL Activity
deployments:
- Daily Cache Updates
detections:
- Detect Spike in Network ACL Activity
product:
baselines/baseline_of_s3_bucket_deletion_activity_by_arn.yml
-2
View File
@@ -22,8 +22,6 @@ references: []
tags:
analytic_story:
- Suspicious AWS S3 Activities
deployments:
- Daily Cache Updates
detections:
- Detect Spike in S3 Bucket deletion
product:
baselines/baseline_of_security_group_activity_by_arn.yml
-2
View File
@@ -23,8 +23,6 @@ references: []
tags:
analytic_story:
- AWS User Monitoring
deployments:
- Daily Cache Updates
detections:
- Detect Spike in Security Group Activity
product:
baselines/baseline_of_smb_traffic___mltk.yml
-2
View File
@@ -40,8 +40,6 @@ tags:
- Hidden Cobra Malware
- Netsh Abuse
- Ransomware
deployments:
- Daily Cache Updates
detections:
- Processes launching netsh
- SMB Traffic Spike - MLTK
baselines/count_of_assets_by_category.yml
-2
View File
@@ -19,8 +19,6 @@ references: []
tags:
analytic_story:
- Asset Tracking
deployments:
- Daily Cache Updates
detections:
- Detect Unauthorized Assets by MAC address
product:
baselines/count_of_unique_ips_connecting_to_ports.yml
+1 -3
View File
@@ -19,9 +19,7 @@ tags:
analytic_story:
- Prohibited Traffic Allowed or Protocol Mismatch
- Ransomware
- Command and Control
deployments:
- Daily Cache Updates
- Command And Control
detections:
- Prohibited Network Traffic Allowed
product:
baselines/create_a_list_of_approved_aws_service_accounts.yml
-2
View File
@@ -21,8 +21,6 @@ references: []
tags:
analytic_story:
- AWS User Monitoring
deployments:
- Daily Cache Updates
detections:
- Detect AWS API Activities From Unapproved Accounts
product:
baselines/deprecated/add_prohibited_processes_to_enterprise_security.yml
-2
View File
@@ -21,8 +21,6 @@ tags:
- Monitor for Unauthorized Software
- SamSam Ransomware
asset_type: Endpoint
deployments:
- Daily Cache Updates
detections:
- Prohibited Software On Endpoint
product:
baselines/deprecated/baseline_of_api_calls_per_user_arn.yml
-2
View File
@@ -22,8 +22,6 @@ references: []
tags:
analytic_story:
- AWS User Monitoring
deployments:
- Daily Cache Updates
detections:
- Detect Spike in AWS API Activity
product:
baselines/deprecated/baseline_of_excessive_aws_instances_launched_by_user___mltk.yml
-2
View File
@@ -33,8 +33,6 @@ tags:
analytic_story:
- AWS Cryptomining
- Suspicious AWS EC2 Activities
deployments:
- Daily Cache Updates
detections:
- Abnormally High AWS Instances Launched by User - MLTK
product:
baselines/deprecated/baseline_of_excessive_aws_instances_terminated_by_user___mltk.yml
-2
View File
@@ -33,8 +33,6 @@ references: []
tags:
analytic_story:
- Suspicious AWS EC2 Activities
deployments:
- Daily Cache Updates
detections:
- Abnormally High AWS Instances Terminated by User - MLTK
product:
baselines/deprecated/previously_seen_api_call_per_user_roles_in_cloudtrail.yml
-2
View File
@@ -22,8 +22,6 @@ references: []
tags:
analytic_story:
- AWS User Monitoring
deployments:
- Daily Cache Updates
detections:
- Detect new API calls from user roles
product:
baselines/deprecated/previously_seen_aws_provisioning_activity_sources.yml
-2
View File
@@ -20,8 +20,6 @@ references: []
tags:
analytic_story:
- AWS Suspicious Provisioning Activities
deployments:
- Daily Cache Updates
detections:
- AWS Cloud Provisioning From Previously Unseen IP Address
- AWS Cloud Provisioning From Previously Unseen City
baselines/deprecated/previously_seen_ec2_amis.yml
-2
View File
@@ -18,8 +18,6 @@ references: []
tags:
analytic_story:
- AWS Cryptomining
deployments:
- Daily Cache Updates
detections:
- EC2 Instance Started With Previously Unseen AMI
product:
baselines/deprecated/previously_seen_ec2_instance_types.yml
-2
View File
@@ -18,8 +18,6 @@ references: []
tags:
analytic_story:
- AWS Cryptomining
deployments:
- Daily Cache Updates
detections:
- EC2 Instance Started With Previously Unseen Instance Type
product:
baselines/deprecated/previously_seen_ec2_launches_by_user.yml
-2
View File
@@ -19,8 +19,6 @@ tags:
analytic_story:
- AWS Cryptomining
- Suspicious AWS EC2 Activities
deployments:
- Daily Cache Updates
detections:
- EC2 Instance Started With Previously Unseen User
product:
baselines/deprecated/previously_seen_users_in_cloudtrail.yml
-2
View File
@@ -23,8 +23,6 @@ references: []
tags:
analytic_story:
- Suspicious AWS Login Activities
deployments:
- Daily Cache Updates
detections:
- Detect AWS Console Login by User from New Country
- Detect AWS Console Login by User from New Region
baselines/deprecated/update_previously_seen_users_in_cloudtrail.yml
-2
View File
@@ -25,8 +25,6 @@ references: []
tags:
analytic_story:
- Suspicious AWS Login Activities
deployments:
- Daily Cache Updates
detections:
- Detect AWS Console Login by User from New Country
- Detect AWS Console Login by User from New Region
baselines/discover_dns_records.yml
-2
View File
@@ -27,8 +27,6 @@ references: []
tags:
analytic_story:
- DNS Hijacking
deployments:
- Daily Cache Updates
detections:
- DNS record changed
product:
baselines/dnstwist_domain_names.yml
-2
View File
@@ -20,8 +20,6 @@ tags:
- Brand Monitoring
- Suspicious Emails
asset_type: Endpoint
deployments:
- Daily Cache Updates
detections:
- Monitor Email For Brand Abuse
- Monitor DNS For Brand Abuse
baselines/identify_systems_creating_remote_desktop_traffic.yml
-2
View File
@@ -21,8 +21,6 @@ tags:
- Ryuk Ransomware
- Hidden Cobra Malware
- Active Directory Lateral Movement
deployments:
- Daily Cache Updates
detections:
- Remote Desktop Network Traffic
product:
baselines/identify_systems_receiving_remote_desktop_traffic.yml
-2
View File
@@ -22,8 +22,6 @@ tags:
- Ryuk Ransomware
- Hidden Cobra Malware
- Active Directory Lateral Movement
deployments:
- Daily Cache Updates
detections:
- Remote Desktop Network Traffic
product:
baselines/identify_systems_using_remote_desktop.yml
-2
View File
@@ -21,8 +21,6 @@ tags:
- Ryuk Ransomware
- Hidden Cobra Malware
- Active Directory Lateral Movement
deployments:
- Daily Cache Updates
detections:
- Remote Desktop Network Traffic
product:
baselines/monitor_successful_backups.yml
-2
View File
@@ -18,8 +18,6 @@ references: []
tags:
analytic_story:
- Monitor Backup Solution
deployments:
- Daily Cache Updates
detections:
- Unsuccessful Netbackup backups
product:
baselines/monitor_unsuccessful_backups.yml
-2
View File
@@ -17,8 +17,6 @@ references: []
tags:
analytic_story:
- Monitor Backup Solution
deployments:
- Daily Cache Updates
detections:
- Unsuccessful Netbackup backups
product:
baselines/previously_seen_aws_cross_account_activity.yml
-2
View File
@@ -22,8 +22,6 @@ references: []
tags:
analytic_story:
- AWS Cross Account Activity
deployments:
- Daily Cache Updates
detections:
- AWS Cross Account Activity From Previously Unseen Account
product:
baselines/previously_seen_aws_cross_account_activity___initial.yml
+6 -2
View File
@@ -26,8 +26,6 @@ references: []
tags:
analytic_story:
- Suspicious Cloud Authentication Activities
deployments:
- 90 Day Baseline
detections:
- AWS Cross Account Activity From Previously Unseen Account
product:
@@ -42,3 +40,9 @@ tags:
- Authentication.src
- Authentication.user_role
security_domain: network
deployment:
scheduling:
cron_schedule: 0 2 * * 0
earliest_time: -90d@d
latest_time: -1d@d
schedule_window: auto
baselines/previously_seen_aws_cross_account_activity___update.yml
-2
View File
@@ -27,8 +27,6 @@ references: []
tags:
analytic_story:
- Suspicious Cloud Authentication Activities
deployments:
- Daily Cache Updates
detections:
- AWS Cross Account Activity From Previously Unseen Account
product:
baselines/previously_seen_aws_regions.yml
-2
View File
@@ -20,8 +20,6 @@ tags:
analytic_story:
- AWS Cryptomining
- Suspicious AWS EC2 Activities
deployments:
- Daily Cache Updates
detections:
- EC2 Instance Started In Previously Unseen Region
product:
baselines/previously_seen_cloud_api_calls_per_user_role___initial.yml
+6 -2
View File
@@ -22,8 +22,6 @@ references: []
tags:
analytic_story:
- Suspicious Cloud User Activities
deployments:
- 90 Day Baseline
detections:
- Cloud API Calls From Previously Unseen User Roles
product:
@@ -37,3 +35,9 @@ tags:
- All_Changes.user
- All_Changes.command
security_domain: network
deployment:
scheduling:
cron_schedule: 0 2 * * 0
earliest_time: -90d@d
latest_time: -1d@d
schedule_window: auto
baselines/previously_seen_cloud_api_calls_per_user_role___update.yml
-2
View File
@@ -24,8 +24,6 @@ references: []
tags:
analytic_story:
- Suspicious Cloud User Activities
deployments:
- Daily Cache Updates
detections:
- Cloud API Calls From Previously Unseen User Roles
product:
baselines/previously_seen_cloud_compute_creations_by_user___initial.yml
+6 -2
View File
@@ -19,8 +19,6 @@ references: []
tags:
analytic_story:
- Cloud Cryptomining
deployments:
- Hourly Cache Updates
detections:
- Cloud Compute Instance Created By Previously Unseen User
product:
@@ -33,3 +31,9 @@ tags:
- All_Changes.object_category
- All_Changes.user
security_domain: network
deployment:
scheduling:
cron_schedule: 55 * * * *
earliest_time: -70m@m
latest_time: -10m@m
schedule_window: auto
baselines/previously_seen_cloud_compute_creations_by_user___update.yml
-2
View File
@@ -22,8 +22,6 @@ references: []
tags:
analytic_story:
- Cloud Cryptomining
deployments:
- Daily Cache Updates
detections:
- Cloud Compute Instance Created By Previously Unseen User
product:
baselines/previously_seen_cloud_compute_images___initial.yml
+6 -2
View File
@@ -21,8 +21,6 @@ references: []
tags:
analytic_story:
- Cloud Cryptomining
deployments:
- 90 Day Baseline
detections:
- Cloud Compute Instance Created With Previously Unseen Image
product:
@@ -34,3 +32,9 @@ tags:
- All_Changes.action
- All_Changes.Instance_Changes.image_id
security_domain: network
deployment:
scheduling:
cron_schedule: 0 2 * * 0
earliest_time: -90d@d
latest_time: -1d@d
schedule_window: auto
baselines/previously_seen_cloud_compute_images___update.yml
-2
View File
@@ -22,8 +22,6 @@ references: []
tags:
analytic_story:
- Cloud Cryptomining
deployments:
- Daily Cache Updates
detections:
- Cloud Compute Instance Created With Previously Unseen Image
product:
baselines/previously_seen_cloud_compute_instance_types___initial.yml
+6 -2
View File
@@ -20,8 +20,6 @@ references: []
tags:
analytic_story:
- Cloud Cryptomining
deployments:
- 90 Day Baseline
detections:
- Cloud Compute Instance Created With Previously Unseen Instance Type
product:
@@ -33,3 +31,9 @@ tags:
- All_Changes.action
- All_Changes.Instance_Changes.instance_type
security_domain: network
deployment:
scheduling:
cron_schedule: 0 2 * * 0
earliest_time: -90d@d
latest_time: -1d@d
schedule_window: auto
baselines/previously_seen_cloud_compute_instance_types___update.yml
-2
View File
@@ -22,8 +22,6 @@ references: []
tags:
analytic_story:
- Cloud Cryptomining
deployments:
- Daily Cache Updates
detections:
- Cloud Compute Instance Created With Previously Unseen Instance Type
product:
baselines/previously_seen_cloud_instance_modifications_by_user___initial.yml
+6 -2
View File
@@ -20,8 +20,6 @@ references: []
tags:
analytic_story:
- Suspicious Cloud Instance Activities
deployments:
- 90 Day Baseline
detections:
- Cloud Instance Modified By Previously Unseen User
product:
@@ -35,3 +33,9 @@ tags:
- All_Changes.status
- All_Changes.user
security_domain: network
deployment:
scheduling:
cron_schedule: 0 2 * * 0
earliest_time: -90d@d
latest_time: -1d@d
schedule_window: auto
baselines/previously_seen_cloud_instance_modifications_by_user___update.yml
-2
View File
@@ -24,8 +24,6 @@ references: []
tags:
analytic_story:
- Suspicious Cloud Instance Activities
deployments:
- Daily Cache Updates
detections:
- Cloud Instance Modified By Previously Unseen User
product:
baselines/previously_seen_cloud_provisioning_activity_sources___initial.yml
+6 -2
View File
@@ -24,8 +24,6 @@ references: []
tags:
analytic_story:
- Suspicious Cloud Provisioning Activities
deployments:
- 90 Day Baseline
detections:
- Cloud Provisioning Activity From Previously Unseen IP Address
- Cloud Provisioning Activity From Previously Unseen City
@@ -41,3 +39,9 @@ tags:
- All_Changes.src
- All_Changes.status
security_domain: network
deployment:
scheduling:
cron_schedule: 0 2 * * 0
earliest_time: -90d@d
latest_time: -1d@d
schedule_window: auto
baselines/previously_seen_cloud_provisioning_activity_sources___update.yml
-2
View File
@@ -29,8 +29,6 @@ references: []
tags:
analytic_story:
- Suspicious Cloud Provisioning Activities
deployments:
- Daily Cache Updates
detections:
- Cloud Provisioning Activity From Previously Unseen IP Address
- Cloud Provisioning Activity From Previously Unseen City
baselines/previously_seen_cloud_regions___initial.yml
+6 -2
View File
@@ -22,8 +22,6 @@ references: []
tags:
analytic_story:
- Cloud Cryptomining
deployments:
- 90 Day Baseline
detections:
- Cloud Compute Instance Created In Previously Unused Region
product:
@@ -35,3 +33,9 @@ tags:
- All_Changes.action
- All_Changes.vendor_region
security_domain: network
deployment:
scheduling:
cron_schedule: 0 2 * * 0
earliest_time: -90d@d
latest_time: -1d@d
schedule_window: auto
baselines/previously_seen_cloud_regions___update.yml
-2
View File
@@ -25,8 +25,6 @@ references: []
tags:
analytic_story:
- Cloud Cryptomining
deployments:
- Daily Cache Updates
detections:
- Cloud Compute Instance Created In Previously Unused Region
product:
baselines/previously_seen_command_line_arguments.yml
-2
View File
@@ -30,8 +30,6 @@ tags:
- Suspicious Command-Line Executions
- Suspicious MSHTA Activity
- IcedID
deployments:
- Daily Cache Updates
detections:
- First time seen command line argument
product:
baselines/previously_seen_ec2_modifications_by_user.yml
-2
View File
@@ -18,8 +18,6 @@ references: []
tags:
analytic_story:
- Unusual AWS EC2 Modifications
deployments:
- Daily Cache Updates
detections:
- EC2 Instance Modified With Previously Unseen User
product:
baselines/previously_seen_running_windows_services___initial.yml
+6 -2
View File
@@ -20,8 +20,6 @@ tags:
- Orangeworm Attack Group
- Windows Service Abuse
- NOBELIUM Group
deployments:
- 90 Day Baseline
detections:
- First Time Seen Running Windows Service
product:
@@ -33,3 +31,9 @@ tags:
- EventCode
- Message
security_domain: endpoint
deployment:
scheduling:
cron_schedule: 0 2 * * 0
earliest_time: -90d@d
latest_time: -1d@d
schedule_window: auto
baselines/previously_seen_running_windows_services___update.yml
+6 -2
View File
@@ -25,8 +25,6 @@ tags:
- Orangeworm Attack Group
- Windows Service Abuse
- NOBELIUM Group
deployments:
- Hourly Cache Updates
detections:
- First Time Seen Running Windows Service
product:
@@ -38,3 +36,9 @@ tags:
- EventCode
- Message
security_domain: endpoint
deployment:
scheduling:
cron_schedule: 55 * * * *
earliest_time: -70m@m
latest_time: -10m@m
schedule_window: auto
baselines/previously_seen_s3_bucket_access_by_remote_ip.yml
-2
View File
@@ -21,8 +21,6 @@ references: []
tags:
analytic_story:
- Suspicious AWS S3 Activities
deployments:
- Daily Cache Updates
detections:
- Detect S3 access from a new IP
product:
baselines/previously_seen_users_in_cloudtrail___initial.yml
+6 -2
View File
@@ -25,8 +25,6 @@ references: []
tags:
analytic_story:
- Suspicious Cloud Authentication Activities
deployments:
- 90 Day Baseline
detections:
- Detect AWS Console Login by User from New Country
- Detect AWS Console Login by User from New Region
@@ -42,3 +40,9 @@ tags:
- Authentication.user
- Authentication.src
security_domain: network
deployment:
scheduling:
cron_schedule: 0 2 * * 0
earliest_time: -90d@d
latest_time: -1d@d
schedule_window: auto
baselines/previously_seen_users_in_cloudtrail___update.yml
-2
View File
@@ -25,8 +25,6 @@ references: []
tags:
analytic_story:
- Suspicious Cloud Authentication Activities
deployments:
- Daily Cache Updates
detections:
- Detect AWS Console Login by User from New Country
- Detect AWS Console Login by User from New Region
baselines/previously_seen_zoom_child_processes___initial.yml
+6 -2
View File
@@ -22,8 +22,6 @@ references: []
tags:
analytic_story:
- Suspicious Zoom Child Processes
deployments:
- 90 Day Baseline
detections:
- First Time Seen Child Process of Zoom
product:
@@ -36,3 +34,9 @@ tags:
- Processes.process_name
- Processes.dest
security_domain: endpoint
deployment:
scheduling:
cron_schedule: 0 2 * * 0
earliest_time: -90d@d
latest_time: -1d@d
schedule_window: auto
baselines/previously_seen_zoom_child_processes___update.yml
+6 -2
View File
@@ -27,8 +27,6 @@ references: []
tags:
analytic_story:
- Suspicious Zoom Child Processes
deployments:
- Hourly Cache Updates
detections:
- First Time Seen Child Process of Zoom
product:
@@ -41,3 +39,9 @@ tags:
- Processes.process_name
- Processes.dest
security_domain: endpoint
deployment:
scheduling:
cron_schedule: 55 * * * *
earliest_time: -70m@m
latest_time: -10m@m
schedule_window: auto
baselines/splunk_command_and_scripting_interpreter_risky_spl_mltk_baseline.yml
+6 -1
View File
@@ -77,4 +77,9 @@ tags:
security_domain: audit
detections:
- Splunk Command and Scripting Interpreter Risky SPL MLTK
deployment:
scheduling:
cron_schedule: 55 * * * *
earliest_time: -70m@m
latest_time: -10m@m
schedule_window: auto
baselines/systems_ready_for_spectre_meltdown_windows_patch.yml
-2
View File
@@ -24,8 +24,6 @@ references: []
tags:
analytic_story:
- Spectre And Meltdown Vulnerabilities
deployments:
- Daily Cache Updates
detections:
- Spectre and Meltdown Vulnerable Systems
product:
baselines/windows_updates_install_failures.yml
-2
View File
@@ -17,8 +17,6 @@ references: []
tags:
analytic_story:
- Monitor for Updates
deployments:
- Daily Cache Updates
detections:
- No Windows Updates in a time frame
product:
baselines/windows_updates_install_successes.yml
-2
View File
@@ -17,8 +17,6 @@ references: []
tags:
analytic_story:
- Monitor for Updates
deployments:
- Daily Cache Updates
detections:
- No Windows Updates in a time frame
product:
bin/contentctl_project/contentctl_core/application/builder/detection_builder.py
+17 -1
View File
@@ -41,7 +41,7 @@ class DetectionBuilder(abc.ABC):
pass
@abc.abstractmethod
def addUnitTest(self, tests: list) -> None:
def addUnitTest(self) -> None:
pass
@abc.abstractmethod
@@ -68,10 +68,26 @@ class DetectionBuilder(abc.ABC):
def setObject(self, path: str) -> None:
pass
@abc.abstractmethod
def addCIS(self) -> None:
pass
@abc.abstractmethod
def addKillChainPhase(self) -> None:
pass
@abc.abstractmethod
def addNist(self) -> None:
pass
@abc.abstractmethod
def reset(self) -> None:
pass
@abc.abstractmethod
def addDatamodel(self) -> None:
pass
@abc.abstractmethod
def getObject(self) -> SecurityContentObject:
pass
bin/contentctl_project/contentctl_core/application/builder/director.py
+1 -1
View File
@@ -11,7 +11,7 @@ from bin.contentctl_project.contentctl_core.domain.entities.enums.enums import S
class Director(abc.ABC):
@abc.abstractmethod
def constructDetection(self, builder: DetectionBuilder, path: str, deployments: list, playbooks: list, baselines: list, tests: list, attack_enrichment: dict, macros: list) -> None:
def constructDetection(self, builder: DetectionBuilder, path: str, deployments: list, playbooks: list, baselines: list, attack_enrichment: dict, macros: list) -> None:
pass
@abc.abstractmethod
bin/contentctl_project/contentctl_core/application/factory/ba_factory.py
+35 -45
View File
@@ -25,11 +25,11 @@ class BAFactoryInputDto:
basic_builder: BasicBuilder
detection_builder: DetectionBuilder
director: Director
attack_enrichment: dict
@dataclass(frozen=True)
class BAFactoryOutputDto:
detections: list
tests: list
class BAFactory():
input_dto: BAFactoryInputDto
@@ -42,9 +42,8 @@ class BAFactory():
def execute(self, input_dto: BAFactoryInputDto) -> None:
self.input_dto = input_dto
print("Creating Security Content - SSA. This may take some time...")
validation_errors = self.createSecurityContent(SecurityContentType.unit_tests)
validation_errors.extend(self.createSecurityContent(SecurityContentType.detections))
validation_errors.extend(Utils.check_ids_for_duplicates(self.ids))
validation_errors = self.createSecurityContent(SecurityContentType.detections)
if len(validation_errors) != 0:
@@ -60,14 +59,11 @@ class BAFactory():
def createSecurityContent(self, type: SecurityContentType) -> list[Tuple[pathlib.Path, ValidationError]]:
objects = []
if type == SecurityContentType.unit_tests:
files = Utils.get_all_yml_files_from_directory(os.path.join(self.input_dto.input_path, 'tests'))
else:
files = Utils.get_all_yml_files_from_directory(os.path.join(self.input_dto.input_path, str(type.name)))
files = Utils.get_all_yml_files_from_directory(os.path.join(self.input_dto.input_path, 'ssa_detections'))
validation_errors:list[Tuple[pathlib.Path, ValidationError]] = []
files_with_ssa = [f for f in files if f.name.startswith('ssa___')]
files_with_ssa = [f for f in files if f.name.startswith('ssa_')]
already_ran = False
progress_percent = 0
@@ -80,49 +76,43 @@ class BAFactory():
progress_percent = ((index+1)/len(files_with_ssa)) * 100
try:
type_string = "UNKNOWN TYPE"
if type == SecurityContentType.detections:
type_string = "Detections"
self.input_dto.director.constructDetection(self.input_dto.detection_builder, file, [], [], [], self.output_dto.tests, {}, [], [])
detection = self.input_dto.detection_builder.getObject()
Utils.add_id(self.ids, detection, file)
#try:
type_string = "UNKNOWN TYPE"
if type == SecurityContentType.detections:
type_string = "Detections"
self.input_dto.director.constructDetection(self.input_dto.detection_builder, file, [], [], [], self.input_dto.attack_enrichment, [], [])
detection = self.input_dto.detection_builder.getObject()
Utils.add_id(self.ids, detection, file)
tag_and_nist_errors = []
if detection.tags.cis20 == None:
error = TypeError(f"Detection Tags missing cis20 field")
tag_and_nist_errors.append(ErrorWrapper(error, loc="cis20"))
tag_and_nist_errors = []
if detection.tags.nist == None:
error = TypeError(f"Detection Tags missing nist field")
tag_and_nist_errors.append(ErrorWrapper(error, loc="nist"))
if detection.tags.cis20 == None:
error = TypeError(f"Detection Tags missing cis20 field")
tag_and_nist_errors.append(ErrorWrapper(error, loc="cis20"))
if detection.tags.nist == None:
error = TypeError(f"Detection Tags missing nist field")
tag_and_nist_errors.append(ErrorWrapper(error, loc="nist"))
if len(tag_and_nist_errors) > 0:
raise ValidationError( tag_and_nist_errors , DetectionTags)
if len(tag_and_nist_errors) > 0:
raise ValidationError( tag_and_nist_errors , DetectionTags)
if not detection.deprecated and not detection.experimental:
self.output_dto.detections.append(detection)
elif type == SecurityContentType.unit_tests:
type_string = "Unit Tests"
self.input_dto.director.constructTest(self.input_dto.basic_builder, str(file))
test = self.input_dto.basic_builder.getObject()
Utils.add_id(self.ids, test, file)
self.output_dto.tests.append(test)
else:
raise(Exception(f"Unsupported content type: [{type}]"))
if detection.status == "production":
self.output_dto.detections.append(detection)
else:
raise(Exception(f"Unsupported content type: [{type}]"))
if (sys.stdout.isatty() and sys.stdin.isatty() and sys.stderr.isatty()) or not already_ran:
already_ran = True
print(f"\r{f'{type_string} Progress'.rjust(23)}: [{progress_percent:3.0f}%]...", end="", flush=True)
if (sys.stdout.isatty() and sys.stdin.isatty() and sys.stderr.isatty()) or not already_ran:
already_ran = True
print(f"\r{f'{type_string} Progress'.rjust(23)}: [{progress_percent:3.0f}%]...", end="", flush=True)
except ValidationError as e:
validation_errors.append((pathlib.Path(file), e))
except Exception as e:
print(f"Unknown exception caught while Creating BA Security Content: {str(e)}")
sys.exit(1)
# except ValidationError as e:
# validation_errors.append((pathlib.Path(file), e))
# except Exception as e:
# print(f"Unknown exception caught while Creating BA Security Content: {str(e)}")
# sys.exit(1)
bin/contentctl_project/contentctl_core/application/factory/factory.py
+5 -20
View File
@@ -42,9 +42,7 @@ class FactoryOutputDto:
deployments: list
macros: list
lookups: list
tests: list
class Factory():
input_dto: FactoryInputDto
@@ -54,16 +52,12 @@ class Factory():
def __init__(self, output_dto: FactoryOutputDto) -> None:
self.output_dto = output_dto
def execute(self, input_dto: FactoryInputDto) -> None:
self.input_dto = input_dto
print("Creating Security Content - ESCU. This may take some time...")
#Accumulate any validation errors that may occur while creating security_contnet
validation_errors = []
# order matters to load and enrich security content types
validation_errors.extend(self.createSecurityContent(SecurityContentType.unit_tests))
validation_errors.extend(self.createSecurityContent(SecurityContentType.lookups))
validation_errors.extend(self.createSecurityContent(SecurityContentType.macros))
validation_errors.extend(self.createSecurityContent(SecurityContentType.deployments))
@@ -87,8 +81,8 @@ class Factory():
objects = []
if type == SecurityContentType.deployments:
files = Utils.get_all_yml_files_from_directory(os.path.join(self.input_dto.input_path, str(type.name)))
elif type == SecurityContentType.unit_tests:
files = Utils.get_all_yml_files_from_directory(os.path.join(self.input_dto.input_path, 'tests'))
elif type == SecurityContentType.detections:
files = Utils.get_all_yml_files_from_directory(os.path.join(self.input_dto.input_path, 'detections'))
else:
files = Utils.get_all_yml_files_from_directory(os.path.join(self.input_dto.input_path, str(type.name)))
@@ -105,7 +99,6 @@ class Factory():
#Non threaded, production version of the construction code
files_without_ssa = [f for f in files if not f.name.startswith('ssa___')]
for index,file in enumerate(files_without_ssa):
#Index + 1 because we are zero indexed, not 1 indexed. This ensures
# that printouts end at 100%, not some other number
progress_percent = ((index+1)/len(files_without_ssa)) * 100
@@ -165,18 +158,11 @@ class Factory():
type_string = "Detections"
self.input_dto.director.constructDetection(self.input_dto.detection_builder, file,
self.output_dto.deployments, self.output_dto.playbooks, self.output_dto.baselines,
self.output_dto.tests, self.input_dto.attack_enrichment, self.output_dto.macros,
self.input_dto.attack_enrichment, self.output_dto.macros,
self.output_dto.lookups, self.input_dto.force_cached_or_offline)
detection = self.input_dto.detection_builder.getObject()
Utils.add_id(self.ids, detection, file)
self.output_dto.detections.append(detection)
elif type == SecurityContentType.unit_tests:
type_string = "Unit Tests"
self.input_dto.director.constructTest(self.input_dto.basic_builder, str(file))
test = self.input_dto.basic_builder.getObject()
Utils.add_id(self.ids, test, file)
self.output_dto.tests.append(test)
else:
raise Exception(f"Unsupported type: [{type}]")
@@ -188,8 +174,7 @@ class Factory():
except ValidationError as e:
validation_errors.append((pathlib.Path(file), e))
except Exception as e:
print(f"Unknown exception caught while Creating Security Content: {str(e)}")
sys.exit(1)
validation_errors.append((pathlib.Path(file), e))
@@ -198,4 +183,4 @@ class Factory():
print(f"\r{f'{type_string} Progress'.rjust(23)}: [{progress_percent:3.0f}%]...", end="", flush=True)
print("Done!")
return validation_errors
return validation_errors
bin/contentctl_project/contentctl_core/application/factory/new_content_factory.py
+14 -13
View File
@@ -36,13 +36,9 @@ class NewContentFactory():
self.output_dto.obj['version'] = 1
self.output_dto.obj['date'] = datetime.today().strftime('%Y-%m-%d')
self.output_dto.obj['author'] = answers['detection_author']
self.output_dto.obj['status'] = 'production'
self.output_dto.obj['type'] = answers['detection_type']
self.output_dto.obj['datamodel'] = answers['datamodels']
if answers['detection_product'] == 'SSA':
answers['datamodels'] = [d.replace(' (SSA)', '') for d in answers['datamodels']]
self.output_dto.obj['datamodel'] = answers['datamodels']
if answers['detection_product'] == 'ESCU':
self.output_dto.obj['datamodel'] = answers['datamodels']
self.output_dto.obj['data_source'] = ['UPDATE_DATA_SOURCE']
self.output_dto.obj['description'] = 'UPDATE_DESCRIPTION'
if answers['detection_product'] == 'ESCU':
file_name = self.output_dto.obj['name'].replace(' ', '_').replace('-','_').replace('.','_').replace('/','_').lower()
@@ -53,15 +49,12 @@ class NewContentFactory():
self.output_dto.obj['tags'] = dict()
self.output_dto.obj['tags']['analytic_story'] = ['UPDATE_STORY_NAME']
self.output_dto.obj['tags']['asset_type'] = 'UPDATE asset_type'
self.output_dto.obj['tags']['cis20'] = ['CIS 3', 'CIS 5', 'CIS 16']
self.output_dto.obj['tags']['atomic_guid'] = ['UPDATE atomic_guid']
self.output_dto.obj['tags']['confidence'] = 'UPDATE value between 1-100'
self.output_dto.obj['tags']['context'] = ['Update context']
self.output_dto.obj['tags']['dataset'] = ['UPDATE_DATASET_URL']
self.output_dto.obj['tags']['drilldown_search'] = ['Add drilldown search']
self.output_dto.obj['tags']['impact'] = 'UPDATE value between 1-100'
self.output_dto.obj['tags']['kill_chain_phases'] = answers['kill_chain_phases']
self.output_dto.obj['tags']['message'] = 'UPDATE message'
self.output_dto.obj['tags']['mitre_attack_id'] = [x.strip() for x in answers['mitre_attack_ids'].split(',')]
self.output_dto.obj['tags']['nist'] = ['DE.CM']
self.output_dto.obj['tags']['observable'] = [{'name': 'UPDATE', 'type': 'UPDATE', 'role': ['UPDATE']}]
if answers['detection_product'] == 'SSA':
self.output_dto.obj['tags']['risk_severity'] = 'UPDATE: <low>, <medium>, <high>'
@@ -69,11 +62,19 @@ class NewContentFactory():
self.output_dto.obj['tags']['product'] = ['Splunk Enterprise','Splunk Enterprise Security','Splunk Cloud']
if answers['detection_product'] == 'SSA':
self.output_dto.obj['tags']['product'] = ['Splunk Behavioral Analytics']
self.output_dto.obj['tags']['required_fields'] = ['UPDATE']
self.output_dto.obj['tags']['risk_score'] = 'UPDATE (impact * confidence)/100'
self.output_dto.obj['tags']['security_domain'] = answers['security_domain']
self.output_dto.obj['source'] = answers['detection_kind']
self.output_dto.obj['tests'] = list()
true_positive_test = dict()
true_positive_test["name"] = 'True Positive Test'
true_positive_test["attack_data"] = [{
"data": "UPDATE url to dataset",
"source": "UPDATE source",
"sourcetype": "UPDATE sourcetype"
}]
self.output_dto.obj['tests'].append(true_positive_test)
elif input_dto.type == SecurityContentType.stories:
questions = NewContentQuestions.get_questions_story()
bin/contentctl_project/contentctl_core/application/factory/utils/new_content_questions.py
+1 -1
View File
@@ -149,7 +149,7 @@ class NewContentQuestions():
'Unauthorized Software',
'Best Practices',
'Cloud Security',
'Command and Control',
'Command And Control',
'Lateral Movement',
'Ransomware',
'Privilege Escalation'
bin/contentctl_project/contentctl_core/application/use_cases/content_changer.py
+2 -2
View File
@@ -90,7 +90,7 @@ class ContentChanger:
def fix_wrong_kill_chain_phases(self, objects : list) -> None:
valid_kill_chain_phases = [
'Reconnaissance', 'Weaponization', 'Delivery',
'Exploitation', 'Installation', 'Command and Control',
'Exploitation', 'Installation', 'Command And Control',
'Actions on Objectives']
for obj in objects:
if 'kill_chain_phases' in obj['tags']:
@@ -196,4 +196,4 @@ class ContentChanger:
for obj in objects:
if 'Command & Control' in obj['tags']['analytic_story']:
obj['tags']['analytic_story'].remove('Command & Control')
obj['tags']['analytic_story'].append('Command and Control')
obj['tags']['analytic_story'].append('Command And Control')
bin/contentctl_project/contentctl_core/application/use_cases/convert.py
+26
View File
@@ -0,0 +1,26 @@
import sys
import shutil
import os
from dataclasses import dataclass
from bin.contentctl_project.contentctl_infrastructure.builder.sigma_converter import SigmaConverter, SigmaConverterInputDto, SigmaConverterOutputDto
from bin.contentctl_project.contentctl_infrastructure.adapter.yml_output import YmlOutput
@dataclass(frozen=True)
class ConvertInputDto:
sigma_converter_input_dto: SigmaConverterInputDto
output_path : str
class Convert:
def execute(self, input_dto: ConvertInputDto) -> None:
sigma_converter_output_dto = SigmaConverterOutputDto([])
sigma_converter = SigmaConverter(sigma_converter_output_dto)
sigma_converter.execute(input_dto.sigma_converter_input_dto)
yml_output = YmlOutput()
yml_output.writeDetections(sigma_converter_output_dto.detections, input_dto.output_path)
bin/contentctl_project/contentctl_core/application/use_cases/doc_gen.py
+1 -1
View File
@@ -16,7 +16,7 @@ class DocGenInputDto:
class DocGen:
def execute(self, input_dto: DocGenInputDto) -> None:
factory_output_dto = FactoryOutputDto([],[],[],[],[],[],[],[],[])
factory_output_dto = FactoryOutputDto([],[],[],[],[],[],[],[])
factory = Factory(factory_output_dto)
factory.execute(input_dto.factory_input_dto)
bin/contentctl_project/contentctl_core/application/use_cases/generate.py
+3 -3
View File
@@ -25,7 +25,7 @@ class Generate:
def execute(self, input_dto: GenerateInputDto) -> None:
if input_dto.product == SecurityContentProduct.ESCU:
factory_output_dto = FactoryOutputDto([],[],[],[],[],[],[],[],[])
factory_output_dto = FactoryOutputDto([],[],[],[],[],[],[],[])
factory = Factory(factory_output_dto)
factory.execute(input_dto.factory_input_dto)
input_dto.adapter.writeHeaders(input_dto.output_path)
@@ -41,13 +41,13 @@ class Generate:
shutil.rmtree(input_dto.output_path + '/complex/', ignore_errors=True)
os.makedirs(input_dto.output_path + '/complex/')
os.makedirs(input_dto.output_path + '/srs/')
factory_output_dto = BAFactoryOutputDto([],[])
factory_output_dto = BAFactoryOutputDto([])
factory = BAFactory(factory_output_dto)
factory.execute(input_dto.ba_factory_input_dto)
input_dto.adapter.writeObjects(factory_output_dto.detections, input_dto.output_path)
elif input_dto.product == SecurityContentProduct.API:
factory_output_dto = FactoryOutputDto([],[],[],[],[],[],[],[],[])
factory_output_dto = FactoryOutputDto([],[],[],[],[],[],[],[])
factory = Factory(factory_output_dto)
factory.execute(input_dto.factory_input_dto)
input_dto.adapter.writeObjects(factory_output_dto.detections, input_dto.output_path, SecurityContentType.detections)
bin/contentctl_project/contentctl_core/application/use_cases/reporting.py
+1 -1
View File
@@ -16,7 +16,7 @@ class ReportingInputDto:
class Reporting:
def execute(self, input_dto: ReportingInputDto) -> None:
factory_output_dto = FactoryOutputDto([],[],[],[],[],[],[],[],[])
factory_output_dto = FactoryOutputDto([],[],[],[],[],[],[],[])
factory = Factory(factory_output_dto)
factory.execute(input_dto.factory_input_dto)
bin/contentctl_project/contentctl_core/application/use_cases/validate.py
+3 -16
View File
@@ -21,30 +21,17 @@ class Validate:
def execute(self, input_dto: ValidateInputDto) -> None:
if input_dto.product == SecurityContentProduct.ESCU:
factory_output_dto = FactoryOutputDto([],[],[],[],[],[],[],[],[])
factory_output_dto = FactoryOutputDto([],[],[],[],[],[],[],[])
factory = Factory(factory_output_dto)
factory.execute(input_dto.factory_input_dto)
elif input_dto.product == SecurityContentProduct.SSA:
factory_output_dto = BAFactoryOutputDto([],[])
factory_output_dto = BAFactoryOutputDto([])
factory = BAFactory(factory_output_dto)
factory.execute(input_dto.ba_factory_input_dto)
# validate detections
# validate tests
self.validate_detection_exist_for_test(factory_output_dto.tests, factory_output_dto.detections)
print('Validation of security content successful.')
def validate_detection_exist_for_test(self, tests : list, detections: list):
for test in tests:
found_detection = False
for detection in detections:
if test.tests[0].file in detection.file_path:
found_detection = True
if not found_detection:
ValueError("detection doesn't exist for test file: " + test.name)
bin/contentctl_project/contentctl_core/domain/constants/constants.py
+56 -19
View File
@@ -1,4 +1,21 @@
ATTACK_TACTICS_KILLCHAIN_MAPPING = {
"Reconnaissance": "Reconnaissance",
"Resource Development": "Weaponization",
"Initial Access": "Delivery",
"Execution": "Installation",
"Persistence": "Installation",
"Privilege Escalation": "Exploitation",
"Defense Evasion": "Exploitation",
"Credential Access": "Exploitation",
"Discovery": "Exploitation",
"Lateral Movement": "Exploitation",
"Collection": "Exploitation",
"Command And Control": "Command And Control",
"Command And Control": "Command And Control",
"Exfiltration": "Actions on Objectives",
"Impact": "Actions on Objectives"
}
SES_CONTEXT_MAPPING = {
"Unknown": 0,
@@ -57,7 +74,7 @@ SES_KILL_CHAIN_MAPPINGS = {
"Delivery": 3,
"Exploitation": 4,
"Installation": 5,
"Command & Control": 6,
"Command And Control": 6,
"Actions on Objectives": 7
}
@@ -76,23 +93,43 @@ SES_OBSERVABLE_ROLE_MAPPING = {
}
SES_OBSERVABLE_TYPE_MAPPING = {
"Other": -1,
"Unknown": 0,
"Device": 1,
"Container": 2,
"Endpoint": 3,
"Hostname": 4,
"IP Address": 5,
"User": 6,
"Username": 7,
"Email": 8,
"Email Address": 9,
"URL": 10,
"URL Domain": 11,
"File": 12,
"File Name": 13,
"File Hash": 14,
"Process": 15,
"Process Name": 16,
"Location": 17
"Hostname": 1,
"IP Address": 2,
"MAC Address": 3,
"User Name": 4,
"Email Address": 5,
"URL String": 6,
"File Name": 7,
"File Hash": 8,
"Process Name": 9,
"Ressource UID": 10,
"Endpoint": 20,
"User": 21,
"Email": 22,
"Uniform Resource Locator": 23,
"File": 24,
"Process": 25,
"Geo Location": 26,
"Container": 27,
"Registry Key": 28,
"Registry Value": 29,
"Other": 99
}
SES_ATTACK_TACTICS_ID_MAPPING = {
"Reconnaissance": "TA0043",
"Resource_Development": "TA0042",
"Initial_Access": "TA0001",
"Execution": "TA0002",
"Persistence": "TA0003",
"Privilege_Escalation": "TA0004",
"Defense_Evasion": "TA0005",
"Credential_Access": "TA0006",
"Discovery": "TA0007",
"Lateral_Movement": "TA0008",
"Collection": "TA0009",
"Command_and_Control": "TA0011",
"Exfiltration": "TA0010",
"Impact": "TA0040"
}
bin/contentctl_project/contentctl_core/domain/entities/data_source.py
+23
View File
@@ -0,0 +1,23 @@
from pydantic import BaseModel, validator, ValidationError
from dataclasses import dataclass
class DataSource(BaseModel):
name: str
id: str
date: str
author: str
type: str
source: str
sourcetype: str
category: str = None
product: str
service: str = None
supported_TA: list
references: list
raw_fields: list
field_mappings: list = None
convert_to_log_source: list = None
bin/contentctl_project/contentctl_core/domain/entities/deployment.py
+7 -7
View File
@@ -13,19 +13,19 @@ from bin.contentctl_project.contentctl_core.domain.entities.deployment_rba impor
from bin.contentctl_project.contentctl_core.domain.entities.deployment_slack import DeploymentSlack
from bin.contentctl_project.contentctl_core.domain.entities.deployment_phantom import DeploymentPhantom
class Deployment(BaseModel, SecurityContentObject):
name: str
id: str
date: str
author: str
description: str
class Deployment(BaseModel):
name: str = None
id: str = None
date: str = None
author: str = None
description: str = None
scheduling: DeploymentScheduling = None
email: DeploymentEmail = None
notable: DeploymentNotable = None
rba: DeploymentRBA = None
slack: DeploymentSlack = None
phantom: DeploymentPhantom = None
tags: dict
tags: dict = None
@validator('name')
bin/contentctl_project/contentctl_core/domain/entities/detection.py
+83 -51
View File
@@ -5,26 +5,37 @@ import time
from pydantic import BaseModel, validator, root_validator
from dataclasses import dataclass
from datetime import datetime
from typing import Union
from bin.contentctl_project.contentctl_core.domain.entities.security_content_object import SecurityContentObject
from bin.contentctl_project.contentctl_core.domain.entities.enums.enums import AnalyticsType
from bin.contentctl_project.contentctl_core.domain.entities.security_content_object import (
SecurityContentObject,
)
from bin.contentctl_project.contentctl_core.domain.entities.enums.enums import (
AnalyticsType,
)
from bin.contentctl_project.contentctl_core.domain.entities.enums.enums import DataModel
from bin.contentctl_project.contentctl_core.domain.entities.detection_tags import DetectionTags
from bin.contentctl_project.contentctl_core.domain.entities.enums.enums import (
DetectionStatus,
)
from bin.contentctl_project.contentctl_core.domain.entities.detection_tags import (
DetectionTags,
)
from bin.contentctl_project.contentctl_core.domain.entities.deployment import Deployment
from bin.contentctl_project.contentctl_core.domain.entities.unit_test import UnitTest
from bin.contentctl_project.contentctl_core.domain.entities.macro import Macro
from bin.contentctl_project.contentctl_core.domain.entities.lookup import Lookup
from bin.contentctl_project.contentctl_core.domain.entities.baseline import Baseline
from bin.contentctl_project.contentctl_core.domain.entities.playbook import Playbook
from bin.contentctl_project.contentctl_core.domain.entities.link_validator import LinkValidator
from bin.contentctl_project.contentctl_core.domain.entities.link_validator import (
LinkValidator,
)
from bin.contentctl_project.contentctl_core.domain.entities.deployment import Deployment
import sys
class Detection(BaseModel, SecurityContentObject):
# detection spec
name: str
@@ -33,17 +44,18 @@ class Detection(BaseModel, SecurityContentObject):
date: str
author: str
type: str
datamodel: list
status: DetectionStatus
description: str
search: str
data_source: list[str]
search: Union[str, dict]
how_to_implement: str
known_false_positives: str
check_references: bool = False #Validation is done in order, this field must be defined first
references: list
tags: DetectionTags
tests: list[UnitTest] = None
# enrichments
datamodel: list = None
deprecated: bool = None
experimental: bool = None
deployment: Deployment = None
@@ -52,7 +64,7 @@ class Detection(BaseModel, SecurityContentObject):
playbooks: list[Playbook] = None
baselines: list[Baseline] = None
mappings: dict = None
test: UnitTest = None
test: Union[UnitTest, dict] = None
macros: list[Macro] = None
lookups: list[Lookup] = None
cve_enrichment: list = None
@@ -61,84 +73,104 @@ class Detection(BaseModel, SecurityContentObject):
source: str = None
nes_fields: str = None
providing_technologies: list = None
runtime: str = None
# @validator('name')
# def name_max_length(cls, v, values):
# @validator('name')v
# def name_max_length(cls, v, values):
# if len(v) > 67:
# raise ValueError('name is longer then 67 chars: ' + v)
# return v
@validator('name')
class Config:
use_enum_values = True
@validator("name")
def name_invalid_chars(cls, v):
invalidChars = set(string.punctuation.replace("-", ""))
if any(char in invalidChars for char in v):
raise ValueError('invalid chars used in name: ' + v)
raise ValueError("invalid chars used in name: " + v)
return v
@validator('id')
@validator("id")
def id_check(cls, v, values):
try:
uuid.UUID(str(v))
except:
raise ValueError('uuid is not valid: ' + values["name"])
raise ValueError("uuid is not valid: " + values["name"])
return v
@validator('date')
@validator("date")
def date_valid(cls, v, values):
try:
datetime.strptime(v, "%Y-%m-%d")
except:
raise ValueError('date is not in format YYYY-MM-DD: ' + values["name"])
raise ValueError("date is not in format YYYY-MM-DD: " + values["name"])
return v
@validator('type')
@validator("type")
def type_valid(cls, v, values):
if v.lower() not in [el.name.lower() for el in AnalyticsType]:
raise ValueError('not valid analytics type: ' + values["name"])
raise ValueError("not valid analytics type: " + values["name"])
return v
@validator('datamodel')
def datamodel_valid(cls, v, values):
for datamodel in v:
if datamodel not in [el.name for el in DataModel]:
raise ValueError('not valid data model: ' + values["name"])
return v
@validator('description', 'how_to_implement')
@validator("description", "how_to_implement")
def encode_error(cls, v, values, field):
try:
v.encode('ascii')
v.encode("ascii")
except UnicodeEncodeError:
raise ValueError('encoding error in ' + field.name + ': ' + values["name"])
raise ValueError("encoding error in " + field.name + ": " + values["name"])
return v
@root_validator
def search_validation(cls, values):
if 'ssa_' not in values['file_path']:
if not '_filter' in values['search']:
raise ValueError('filter macro missing in: ' + values["name"])
if any(x in values['search'] for x in ['eventtype=', 'sourcetype=', ' source=', 'index=']):
if not 'index=_internal' in values['search']:
raise ValueError('Use source macro instead of eventtype, sourcetype, source or index in detection: ' + values["name"])
return values
# @root_validator
# def search_validation(cls, values):
# if 'ssa_' not in values['file_path']:
# if not '_filter' in values['search']:
# raise ValueError('filter macro missing in: ' + values["name"])
# if any(x in values['search'] for x in ['eventtype=', 'sourcetype=', ' source=', 'index=']):
# if not 'index=_internal' in values['search']:
# raise ValueError('Use source macro instead of eventtype, sourcetype, source or index in detection: ' + values["name"])
# return values
@root_validator
def name_max_length(cls, values):
# Check max length only for ESCU searches, SSA does not have that constraint
if 'ssa_' not in values['file_path']:
if "ssa_" not in values["file_path"]:
if len(values["name"]) > 67:
raise ValueError('name is longer then 67 chars: ' + values["name"])
raise ValueError("name is longer then 67 chars: " + values["name"])
return values
@validator('references')
def references_check(cls, v, values):
return LinkValidator.SecurityContentObject_validate_references(v, values)
# @validator('references')
# def references_check(cls, v, values):
# return LinkValidator.SecurityContentObject_validate_references(v, values)
@validator('search')
@root_validator
def missing_test_file(cls, values):
if values["status"] == DetectionStatus.production:
if "tests" not in values:
raise ValueError("Missing test file for detection: " + values["name"])
return values
@validator("search")
def search_validate(cls, v, values):
# write search validator
return v
@validator("tests")
def tests_validate(cls, v, values):
if values["status"] != DetectionStatus.production and not v:
raise ValueError(
"tests value is needed for production detection: " + values["name"]
)
return v
@validator("experimental", always=True)
def experimental_validate(cls, v, values):
if DetectionStatus(values["status"]) == DetectionStatus.experimental:
return True
return False
@validator("deprecated", always=True)
def deprecated_validate(cls, v, values):
if DetectionStatus(values["status"]) == DetectionStatus.deprecated:
return True
return False
bin/contentctl_project/contentctl_core/domain/entities/detection_tags.py
+23 -15
View File
@@ -4,6 +4,7 @@ from pydantic import BaseModel, validator, ValidationError
from bin.contentctl_project.contentctl_core.domain.entities.mitre_attack_enrichment import MitreAttackEnrichment
from bin.contentctl_project.contentctl_core.domain.constants.constants import *
class DetectionTags(BaseModel):
# detection spec
name: str
@@ -12,21 +13,25 @@ class DetectionTags(BaseModel):
automated_detection_testing: str = None
cis20: list = None
confidence: str
context: list
context: list = None
dataset: list = None
impact: int
kill_chain_phases: list
kill_chain_phases: list = None
message: str
mitre_attack_id: list = None
nist: list = None
observable: list
product: list
required_fields: list
required_fields: list = None
risk_score: int
security_domain: str
risk_severity: str = None
cve: list = None
supported_tas: list = None
atomic_guid: list = None
drilldown_search: str = None
manual_test: str = None
# enrichment
mitre_attack_enrichments: list[MitreAttackEnrichment] = []
@@ -36,9 +41,12 @@ class DetectionTags(BaseModel):
risk_level_id: int = None
risk_level: str = None
observable_str: str = None
evidence_str: str = None
kill_chain_phases_id: list = None
research_site_url: str = None
event_schema: str = None
mappings: list = None
annotations: dict = None
@validator('cis20')
@@ -104,19 +112,19 @@ class DetectionTags(BaseModel):
raise ValueError('Mitre Attack ID are not following the pattern Txxxx: ' + values["name"])
return v
@validator('observable')
def tags_observable(cls,v,values):
valid_roles = SES_OBSERVABLE_ROLE_MAPPING.keys()
valid_types = SES_OBSERVABLE_TYPE_MAPPING.keys()
# @validator('observable')
# def tags_observable(cls,v,values):
# valid_roles = SES_OBSERVABLE_ROLE_MAPPING.keys()
# valid_types = SES_OBSERVABLE_TYPE_MAPPING.keys()
for value in v:
if value['type'] in valid_types:
for role in value['role']:
if role not in valid_roles:
raise ValueError('Observable role ' + role + ' not valid for ' + values["name"] + '. valid options are ' + str(valid_roles))
else:
raise ValueError('Observable type ' + value['type'] + ' not valid for ' + values["name"] + '. valid options are ' + str(valid_types))
return v
# for value in v:
# if value['type'] in valid_types:
# for role in value['role']:
# if role not in valid_roles:
# raise ValueError('Observable role ' + role + ' not valid for ' + values["name"] + '. valid options are ' + str(valid_roles))
# else:
# raise ValueError('Observable type ' + value['type'] + ' not valid for ' + values["name"] + '. valid options are ' + str(valid_types))
# return v
@validator('product')
def tags_product(cls, v, values):
bin/contentctl_project/contentctl_core/domain/entities/enums/enums.py
+16
View File
@@ -7,6 +7,7 @@ class AnalyticsType(enum.Enum):
hunting = 3
correlation = 4
class DataModel(enum.Enum):
Endpoint = 1
Network_Traffic = 2
@@ -26,6 +27,7 @@ class DataModel(enum.Enum):
Risk = 16
Splunk_Audit = 17
class SecurityContentType(enum.Enum):
detections = 1
baselines = 2
@@ -38,8 +40,22 @@ class SecurityContentType(enum.Enum):
unit_tests = 9
attack_data = 10
class SecurityContentProduct(enum.Enum):
ESCU = 1
SSA = 2
API = 3
CUSTOM = 4
class SigmaConverterTarget(enum.Enum):
CIM = 1
RAW = 2
OCSF = 3
ALL = 4
class DetectionStatus(enum.Enum):
production = "production"
deprecated = "deprecated"
experimental = "experimental"
bin/contentctl_project/contentctl_core/domain/entities/unit_test.py
+5 -3
View File
@@ -3,9 +3,11 @@
from pydantic import BaseModel, validator, ValidationError
from bin.contentctl_project.contentctl_core.domain.entities.security_content_object import SecurityContentObject
from bin.contentctl_project.contentctl_core.domain.entities.unit_test_test import UnitTestTest
from bin.contentctl_project.contentctl_core.domain.entities.unit_test_attack_data import UnitTestAttackData
from bin.contentctl_project.contentctl_core.domain.entities.unit_test_baseline import UnitTestBaseline
class UnitTest(BaseModel, SecurityContentObject):
class UnitTest(BaseModel):
name: str
tests: list[UnitTestTest]
baselines: list[UnitTestBaseline] = None
attack_data: list[UnitTestAttackData]
bin/contentctl_project/contentctl_core/domain/entities/unit_test_attack_data.py
+3 -3
View File
@@ -4,8 +4,8 @@ from pydantic import BaseModel, validator, ValidationError
class UnitTestAttackData(BaseModel):
file_name: str
data: str
source: str
file_name: str = None
data: str = None
source: str = None
sourcetype: str = None
update_timestamp: bool = None
bin/contentctl_project/contentctl_core/domain/entities/unit_test_old.py
+12
View File
@@ -0,0 +1,12 @@
from pydantic import BaseModel, validator, ValidationError
from bin.contentctl_project.contentctl_core.domain.entities.unit_test import UnitTest
class UnitTestOld(BaseModel):
name: str
tests: list[UnitTest]
bin/contentctl_project/contentctl_infrastructure/adapter/finding_report_writer.py
+40 -34
View File
@@ -10,31 +10,32 @@ class FindingReportObject():
@staticmethod
def writeFindingReport(detection : Detection) -> None:
if detection.tags.confidence < 33:
detection.tags.confidence_id = 1
elif detection.tags.confidence < 66:
detection.tags.confidence_id = 2
else:
detection.tags.confidence_id = 3
# if detection.tags.confidence < 33:
# detection.tags.confidence_id = 1
# elif detection.tags.confidence < 66:
# detection.tags.confidence_id = 2
# else:
# detection.tags.confidence_id = 3
detection.tags.context_ids = list()
for context in detection.tags.context:
detection.tags.context_ids.append(SES_CONTEXT_MAPPING[context])
# detection.tags.context_ids = list()
# for context in detection.tags.context:
# detection.tags.context_ids.append(SES_CONTEXT_MAPPING[context])
if detection.tags.impact < 20:
detection.tags.impact_id = 1
elif detection.tags.impact < 40:
detection.tags.impact_id = 2
elif detection.tags.impact < 60:
detection.tags.impact_id = 3
elif detection.tags.impact < 80:
detection.tags.impact_id = 4
else:
detection.tags.impact_id = 5
# if detection.tags.impact < 20:
# detection.tags.impact_id = 1
# elif detection.tags.impact < 40:
# detection.tags.impact_id = 2
# elif detection.tags.impact < 60:
# detection.tags.impact_id = 3
# elif detection.tags.impact < 80:
# detection.tags.impact_id = 4
# else:
# detection.tags.impact_id = 5
# detection.tags.kill_chain_phases_id = dict()
# for kill_chain_phase in detection.tags.kill_chain_phases:
# detection.tags.kill_chain_phases_id[kill_chain_phase] = SES_KILL_CHAIN_MAPPINGS[kill_chain_phase]
detection.tags.kill_chain_phases_id = list()
for kill_chain_phase in detection.tags.kill_chain_phases:
detection.tags.kill_chain_phases_id.append(SES_KILL_CHAIN_MAPPINGS[kill_chain_phase])
if detection.tags.risk_score < 20:
detection.tags.risk_level_id = 0
@@ -52,23 +53,28 @@ class FindingReportObject():
detection.tags.risk_level_id = 4
detection.tags.risk_level = "Critical"
observable_str = "["
for i in range(len(detection.tags.observable)):
role_list = []
for role in detection.tags.observable[i]["role"]:
role_list.append(str(SES_OBSERVABLE_ROLE_MAPPING[role]))
observable_str = observable_str + 'create_map("name", "' + detection.tags.observable[i]["name"] + '", "role_ids", [' + ",".join(role_list) + '], "type_id", ' + str(SES_OBSERVABLE_TYPE_MAPPING[detection.tags.observable[i]["type"]]) + ', "value", ' + detection.tags.observable[i]["name"] + ')'
if not i == len(detection.tags.observable):
observable_str = observable_str + ', '
observable_str = observable_str + ']'
# observable_str = "["
# for i in range(len(detection.tags.observable)):
# observable_str = observable_str + 'create_map("name", "' + detection.tags.observable[i]["name"] + '", "type_id", ' + str(SES_OBSERVABLE_TYPE_MAPPING[detection.tags.observable[i]["type"]]) + ', "value", ' + detection.tags.observable[i]["name"].replace(".", "_") + ')'
# if not i == (len(detection.tags.observable) - 1):
# observable_str = observable_str + ', '
# observable_str = observable_str + ']'
detection.tags.observable_str = observable_str
# detection.tags.observable_str = observable_str
evidence_str = "create_map("
for i in range(len(detection.tags.observable)):
evidence_str = evidence_str + '"' + detection.tags.observable[i]["name"] + '", ' + detection.tags.observable[i]["name"].replace(".", "_")
if not i == (len(detection.tags.observable) - 1):
evidence_str = evidence_str + ', '
evidence_str = evidence_str + ')'
detection.tags.evidence_str = evidence_str
j2_env = Environment(
loader=FileSystemLoader(os.path.join(os.path.dirname(__file__), 'templates')),
trim_blocks=True)
template = j2_env.get_template('finding_report.j2')
body = template.render(detection=detection)
body = template.render(detection=detection, attack_tactics_id_mapping=SES_ATTACK_TACTICS_ID_MAPPING)
return body
bin/contentctl_project/contentctl_infrastructure/adapter/obj_to_svg_adapter.py
+2 -2
View File
@@ -17,10 +17,10 @@ class ObjToSvgAdapter(Adapter):
obj = dict()
for detection in detections_tmp:
if not detection.deprecated:
if not (detection.status == "deprecated"):
detections.append(detection)
if not detection.test and not detection.experimental:
if not detection.test and not (detection.status == "experimental"):
detection_without_test = detection_without_test + 1
bin/contentctl_project/contentctl_infrastructure/adapter/obj_to_yml_adapter.py
+41 -29
View File
@@ -1,10 +1,14 @@
import os
import re
from urllib.parse import urlparse
from bin.contentctl_project.contentctl_infrastructure.adapter.yml_writer import YmlWriter
from bin.contentctl_project.contentctl_core.application.adapter.adapter import Adapter
from bin.contentctl_project.contentctl_core.domain.entities.enums.enums import SecurityContentType
from bin.contentctl_project.contentctl_infrastructure.adapter.finding_report_writer import FindingReportObject
from bin.contentctl_project.contentctl_core.domain.entities.unit_test_old import UnitTestOld
class ObjToYmlAdapter(Adapter):
input_path: str
@@ -36,29 +40,58 @@ class ObjToYmlAdapter(Adapter):
obj.tags.research_site_url = research_site_url
# add ocsf schema tag
obj.tags.event_schema = 'cim'
obj.tags.event_schema = 'ocsf'
body = FindingReportObject.writeFindingReport(obj)
if obj.test:
test_dict = {
"name": obj.name + " Unit Test",
"tests": [obj.test.dict()]
}
test_dict["tests"][0]["name"] = obj.name
for count in range(len(test_dict["tests"][0]["attack_data"])):
a = urlparse(test_dict["tests"][0]["attack_data"][count]["data"])
test_dict["tests"][0]["attack_data"][count]["file_name"] = os.path.basename(a.path)
test = UnitTestOld.parse_obj(test_dict)
obj.test = test
# create annotations object
obj.tags.annotations = {
"analytic_story": obj.tags.analytic_story,
"cis20": obj.tags.cis20,
"kill_chain_phases": obj.tags.kill_chain_phases,
"mitre_attack_id": obj.tags.mitre_attack_id,
"nist": obj.tags.nist
}
obj.runtime = "SPL-DSP"
# remove unncessary fields
YmlWriter.writeYmlFile(file_path, obj.dict(
exclude_none=True,
include =
{
"name": True,
"id": True,
"eventSchema": True,
"version": True,
"description": True,
"search": True,
"how_to_implement": True,
"known_false_positives": True,
"references": True,
"runtime": True,
"tags":
{
"analytic_story": True,
"cis20" : True,
"nist": True,
"kill_chain_phases": True,
"mitre_attack_id": True,
#"analytic_story": True,
#"cis20" : True,
#"nist": True,
#"kill_chain_phases": True,
"annotations": True,
"mappings": True,
#"mitre_attack_id": True,
"risk_severity": True,
"risk_score": True,
"security_domain": True,
@@ -91,37 +124,16 @@ class ObjToYmlAdapter(Adapter):
# Add Finding Report Object
with open(file_path, 'r') as file:
data = file.read().replace('--body--', body)
data = file.read().replace('--finding_report--', body)
f = open(file_path, "w")
f.write(data)
f.close()
def writeObjectNewContent(self, object: dict, type: SecurityContentType) -> None:
if type == SecurityContentType.detections:
file_path = os.path.join(self.input_path, 'detections', object['source'], self.convertNameToFileName(object['name'],object['tags']['product']))
test_obj = {}
test_obj['name'] = object['name'] + ' Unit Test'
test_obj['tests'] = [
{
'name': object['name'],
'file': object['source'] + '/' + self.convertNameToFileName(object['name'],object['tags']['product']),
'pass_condition': '| stats count | where count > 0',
'earliest_time': '-24h',
'latest_time': 'now',
'attack_data': [
{
'file_name': 'UPDATE',
'data': 'UPDATE',
'source': 'UPDATE',
'sourcetype': 'UPDATE',
'update_timestamp': True
}
]
}
]
file_path_test = os.path.join(self.input_path, 'tests', object['source'], self.convertNameToTestFileName(object['name'],object['tags']['product']))
YmlWriter.writeYmlFile(file_path_test, test_obj)
object.pop('source')
elif type == SecurityContentType.stories:
file_path = os.path.join(self.input_path, 'stories', self.convertNameToFileName(object['name'],object['tags']['product']))
bin/contentctl_project/contentctl_infrastructure/adapter/templates/finding_report.j2
+28 -11
View File
@@ -1,11 +1,28 @@
create_map("category_id", 101, "class_id", 101000, "detection_start_time", start_time,
"detection_end_time", end_time, "device_entities", [create_map("uid", ucast(map_get(input_event, "enrichments.device_entities.device.uid"), "string", null), "type_id", 0)],
"disposition_id", 1, "end_time", end_time, "event_id", 10100001, "event_time", timestamp,
"finding", create_map("confidence", {{ detection.tags.confidence }}, "confidence_id", {{ detection.tags.confidence_id }},
"context_ids", {{ detection.tags.context_ids }}, "impact", {{ detection.tags.impact }}, "impact_id", {{ detection.tags.impact_id }},
"kill_chain_phase", {{ detection.tags.kill_chain_phases[0] }}, "kill_chain_phase_id", {{ detection.tags.kill_chain_phases_id[0] }},
"risk_level", {{ detection.tags.risk_level }}, "risk_level_id", {{ detection.tags.risk_level_id }}, "type_id", 1, "ref_event_uid", event_id),
"message", {{ detection.tags.message }}, "metadata", create_map("log_name", {{ detection.datamodel[0] }}, "version",
"1.0.0"), "observables", {{ detection.tags.observables_str }}, "origin", create_map("product", create_map("name", "Splunk Behavioral Analytics")),
"rule", create_map("name", "{{ detection.name }}", "uid", "{{ detection.id }}", "version", "1"), "start_time", start_time, "time", start_time,
"user_entities", [create_map("uid", ucast(map_get(input_event, "enrichments.user_entities.user.uid"),"string", null))])
| eval body=create_map(
"devices", [
create_map(
"hostname", device_hostname, "type_id", 0, "uuid", ucast(map_get(device,"uuid"), "string", null)
)
],
"time", timestamp,
"evidence", {{ detection.tags.evidence_str }},
"message", concat("{{ detection.name }} has been triggered on ", device_hostname, " by ", process_user_name, "."),
"users", [
create_map(
"name", process_user_name, "uuid", ucast(map_get(process_user,"uuid"), "string", null)
)
],
"activity_id", 1,
"category_uid", 2,
"class_uid", 102001,
"risk_level_id", {{ detection.tags.risk_level_id }},
"risk_score", {{ detection.tags.risk_score }},
"severity_id", 0,
"rule", create_map("name", "{{ detection.name }}", "uid", "{{ detection.id }}", "type", "Streaming"),
"metadata", create_map("customer_uid", ucast(map_get(metadata,"customer_uid"), "string", null), "product", create_map("name", "Behavior Analytics", "vendor_name", "Splunk"), "version", "1.0.0-rc.2", "logged_time", time()),
"type_uid", 10200101,
"start_time", timestamp,
"end_time", timestamp
)
| into write_ba_finding_events();
bin/contentctl_project/contentctl_infrastructure/adapter/templates/savedsearches_detections.j2
+5 -1
View File
@@ -38,7 +38,7 @@ action.escu.providing_technologies = []
{% endif %}
{% if detection.tags.analytic_story is defined %}
action.escu.analytic_story = {{ detection.tags.analytic_story | tojson }}
{% if detection.tags.risk_score is defined and detection.type | lower != "correlation" %}
{% if detection.deployment.rba.enabled is defined %}
action.risk = 1
action.risk.param._risk_message = {{ detection.tags.message }}
action.risk.param._risk = {{ detection.risk | tojson }}
@@ -73,6 +73,10 @@ action.notable.param.rule_description = {{ detection.deployment.notable.rule_des
action.notable.param.rule_title = {% if detection.type | lower == "correlation" %}RBA: {{ detection.deployment.notable.rule_title | custom_jinja2_enrichment_filter(detection) }}{% else %}{{ detection.deployment.notable.rule_title | custom_jinja2_enrichment_filter(detection) }}{% endif +%}
action.notable.param.security_domain = {{ detection.tags.security_domain }}
action.notable.param.severity = high
{% if detection.tags.drilldown_search is not none %}
action.notable.param.drilldown_name = {{ detection.name }}
action.notable.param.drilldown_search = {{ detection.tags.drilldown_search }}
{% endif %}
{% endif %}
{% if detection.deployment.email.to is defined %}
action.email.subject.alert = {{ detection.deployment.email.subject | custom_jinja2_enrichment_filter(detection) }}
bin/contentctl_project/contentctl_infrastructure/adapter/yml_output.py
+65
View File
@@ -0,0 +1,65 @@
import os
from bin.contentctl_project.contentctl_infrastructure.adapter.yml_writer import YmlWriter
class YmlOutput():
def writeDetections(self, objects: list, output_path : str) -> None:
for obj in objects:
file_path = obj.file_path
YmlWriter.writeYmlFile(os.path.join(output_path, file_path), obj.dict(
exclude_none=True,
include =
{
"name": True,
"id": True,
"version": True,
"date": True,
"author": True,
"type": True,
"status": True,
"description": True,
"data_source": True,
"search": True,
"how_to_implement": True,
"known_false_positives": True,
"references": True,
"tags":
{
"analytic_story": True,
"asset_type": True,
"atomic_guid": True,
"confidence": True,
"impact": True,
"drilldown_search": True,
"mappings": True,
"message": True,
"mitre_attack_id": True,
"kill_chain_phases:": True,
"observable": True,
"product": True,
"required_fields": True,
"risk_score": True,
"security_domain": True
},
"tests":
{
'__all__':
{
"name": True,
"attack_data": {
'__all__':
{
"data": True,
"source": True,
"sourcetype": True
}
}
}
}
}
))
bin/contentctl_project/contentctl_infrastructure/builder/backend_splunk_ba.py
+129
View File
@@ -0,0 +1,129 @@
import re
from sigma.conversion.state import ConversionState
from sigma.rule import SigmaRule
from sigma.conversion.base import TextQueryBackend
from sigma.conversion.deferred import DeferredTextQueryExpression
from sigma.conditions import ConditionFieldEqualsValueExpression, ConditionOR, ConditionAND, ConditionNOT, ConditionItem
from sigma.types import SigmaCompareExpression
from sigma.exceptions import SigmaFeatureNotSupportedByBackendError
from sigma.pipelines.splunk.splunk import splunk_sysmon_process_creation_cim_mapping, splunk_windows_registry_cim_mapping, splunk_windows_file_event_cim_mapping
from bin.contentctl_project.contentctl_core.domain.entities.detection import Detection
from typing import ClassVar, Dict, List, Optional, Pattern, Tuple
class SplunkBABackend(TextQueryBackend):
"""Splunk SPL backend."""
precedence: ClassVar[Tuple[ConditionItem, ConditionItem, ConditionItem]] = (ConditionNOT, ConditionOR, ConditionAND)
group_expression : ClassVar[str] = "({expr})"
parenthesize : bool = True
or_token : ClassVar[str] = "OR"
and_token : ClassVar[str] = "AND"
not_token : ClassVar[str] = "NOT"
eq_token : ClassVar[str] = "="
field_quote: ClassVar[str] = '"'
field_quote_pattern: ClassVar[Pattern] = re.compile("^[\w.]+$")
str_quote : ClassVar[str] = '"'
escape_char : ClassVar[str] = "\\"
wildcard_multi : ClassVar[str] = "%"
wildcard_single : ClassVar[str] = "%"
add_escaped : ClassVar[str] = "\\"
re_expression : ClassVar[str] = "match_regex({field}, /(?i){regex}/)=true"
re_escape_char : ClassVar[str] = "\\"
re_escape : ClassVar[Tuple[str]] = ('"',)
cidr_expression : ClassVar[str] = "{value}"
compare_op_expression : ClassVar[str] = "{field}{operator}{value}"
compare_operators : ClassVar[Dict[SigmaCompareExpression.CompareOperators, str]] = {
SigmaCompareExpression.CompareOperators.LT : "<",
SigmaCompareExpression.CompareOperators.LTE : "<=",
SigmaCompareExpression.CompareOperators.GT : ">",
SigmaCompareExpression.CompareOperators.GTE : ">=",
}
field_null_expression : ClassVar[str] = "{field} IS NOT NULL"
convert_or_as_in : ClassVar[bool] = False
convert_and_as_in : ClassVar[bool] = False
in_expressions_allow_wildcards : ClassVar[bool] = True
field_in_list_expression : ClassVar[str] = "{field} {op} ({list})"
or_in_operator : ClassVar[Optional[str]] = "IN"
list_separator : ClassVar[str] = ", "
unbound_value_str_expression : ClassVar[str] = '{value}'
unbound_value_num_expression : ClassVar[str] = '{value}'
unbound_value_re_expression : ClassVar[str] = '{value}'
deferred_start : ClassVar[str] = " "
deferred_separator : ClassVar[str] = " OR "
deferred_only_query : ClassVar[str] = "*"
wildcard_match_expression : ClassVar[Optional[str]] = "like({field}, {value})"
def __init__(self, processing_pipeline: Optional["sigma.processing.pipeline.ProcessingPipeline"] = None, collect_errors: bool = False, min_time : str = "-30d", max_time : str = "now", detection : Detection = None, field_mapping: dict = None, **kwargs):
super().__init__(processing_pipeline, collect_errors, **kwargs)
self.min_time = min_time or "-30d"
self.max_time = max_time or "now"
self.detection = detection
self.field_mapping = field_mapping
def finalize_query_data_model(self, rule: SigmaRule, query: str, index: int, state: ConversionState) -> str:
try:
fields = state.processing_state["fields"]
except KeyError:
raise SigmaFeatureNotSupportedByBackendError("No fields specified by processing pipeline")
# fields_input_parsing = ''
# for count, value in enumerate(fields):
# fields_input_parsing = fields_input_parsing + value + '=ucast(map_get(input_event, "' + value + '"), "string", null)'
# if not count == len(fields) - 1:
# fields_input_parsing = fields_input_parsing + ', '
detection_str = """
| from read_ba_enriched_events()
| eval timestamp = ucast(map_get(input_event,"time"),"long", null)
| eval metadata = ucast(map_get(input_event, "metadata"),"map<string, any>", null)
| eval metadata_uid = ucast(map_get(metadata, "uid"),"string", null)
""".replace("\n", " ")
parsed_fields = []
for field in self.field_mapping["mapping"].keys():
mapped_field = self.field_mapping["mapping"][field]
parent = 'input_event'
i = 1
values = mapped_field.split('.')
for val in values:
if parent == "input_event":
new_val = val
else:
new_val = parent + '_' + val
if new_val in parsed_fields:
parent = new_val
i = i + 1
continue
if i == len(values):
parser_str = '| eval ' + new_val + '' + '=ucast(map_get(' + parent + ',"' + val + '"), "string", null) '
else:
parser_str = '| eval ' + new_val + '' + '=ucast(map_get(' + parent + ',"' + val + '"), "map<string, any>", null) '
detection_str = detection_str + parser_str
parsed_fields.append(new_val)
parent = new_val
i = i + 1
detection_str = detection_str + "| where " + query
detection_str = detection_str.replace("\\\\\\\\", "\\\\")
return detection_str
def finalize_output_data_model(self, queries: List[str]) -> List[str]:
return queries
bin/contentctl_project/contentctl_infrastructure/builder/security_content_baseline_builder.py
+28 -25
View File
@@ -30,35 +30,38 @@ class SecurityContentBaselineBuilder(BaselineBuilder):
sys.exit(1)
def addDeployment(self, deployments: list) -> None:
matched_deployments = []
for d in deployments:
d_tags = dict(d.tags)
baseline_dict = self.baseline.dict()
baseline_tags_dict = self.baseline.tags.dict()
for d_tag in d_tags.keys():
for attr in baseline_dict.keys():
if attr == d_tag:
if isinstance(baseline_dict[attr], str):
if baseline_dict[attr] == d_tags[d_tag]:
matched_deployments.append(d)
elif isinstance(baseline_dict[attr], list):
if d_tags[d_tag] in baseline_dict[attr]:
matched_deployments.append(d)
if not self.baseline.deployment:
for attr in baseline_tags_dict.keys():
if attr == d_tag:
if isinstance(baseline_tags_dict[attr], str):
if baseline_tags_dict[attr] == d_tags[d_tag]:
matched_deployments.append(d)
elif isinstance(baseline_tags_dict[attr], list):
if d_tags[d_tag] in baseline_tags_dict[attr]:
matched_deployments.append(d)
matched_deployments = []
if len(matched_deployments) == 0:
raise ValueError('No deployment found for baseline: ' + self.baseline.name)
for d in deployments:
d_tags = dict(d.tags)
baseline_dict = self.baseline.dict()
baseline_tags_dict = self.baseline.tags.dict()
for d_tag in d_tags.keys():
for attr in baseline_dict.keys():
if attr == d_tag:
if isinstance(baseline_dict[attr], str):
if baseline_dict[attr] == d_tags[d_tag]:
matched_deployments.append(d)
elif isinstance(baseline_dict[attr], list):
if d_tags[d_tag] in baseline_dict[attr]:
matched_deployments.append(d)
self.baseline.deployment = matched_deployments[-1]
for attr in baseline_tags_dict.keys():
if attr == d_tag:
if isinstance(baseline_tags_dict[attr], str):
if baseline_tags_dict[attr] == d_tags[d_tag]:
matched_deployments.append(d)
elif isinstance(baseline_tags_dict[attr], list):
if d_tags[d_tag] in baseline_tags_dict[attr]:
matched_deployments.append(d)
if len(matched_deployments) == 0:
raise ValueError('No deployment found for baseline: ' + self.baseline.name)
self.baseline.deployment = matched_deployments[-1]
def reset(self) -> None:
bin/contentctl_project/contentctl_infrastructure/builder/security_content_detection_builder.py
+91 -45
View File
@@ -12,7 +12,7 @@ from bin.contentctl_project.contentctl_core.domain.entities.macro import Macro
from bin.contentctl_project.contentctl_core.domain.entities.mitre_attack_enrichment import MitreAttackEnrichment
from bin.contentctl_project.contentctl_infrastructure.builder.cve_enrichment import CveEnrichment
from bin.contentctl_project.contentctl_infrastructure.builder.splunk_app_enrichment import SplunkAppEnrichment
from bin.contentctl_project.contentctl_core.domain.constants.constants import *
class SecurityContentDetectionBuilder(DetectionBuilder):
security_content_obj : SecurityContentObject
@@ -38,31 +38,35 @@ class SecurityContentDetectionBuilder(DetectionBuilder):
def addDeployment(self, deployments: list) -> None:
if self.security_content_obj:
matched_deployments = []
for d in deployments:
d_tags = dict(d.tags)
for d_tag in d_tags.keys():
for attr in dir(self.security_content_obj):
if not (attr.startswith('__') or attr.startswith('_')):
if attr == d_tag:
if type(self.security_content_obj.__getattribute__(attr)) is str:
attr_values = [self.security_content_obj.__getattribute__(attr)]
else:
attr_values = self.security_content_obj.__getattribute__(attr)
for attr_value in attr_values:
if attr_value == d_tags[d_tag]:
matched_deployments.append(d)
if not self.security_content_obj.deployment:
if len(matched_deployments) == 0:
self.security_content_obj.deployment = None
else:
self.security_content_obj.deployment = matched_deployments[-1]
matched_deployments = []
for d in deployments:
d_tags = dict(d.tags)
for d_tag in d_tags.keys():
for attr in dir(self.security_content_obj):
if not (attr.startswith('__') or attr.startswith('_')):
if attr == d_tag:
if type(self.security_content_obj.__getattribute__(attr)) is str:
attr_values = [self.security_content_obj.__getattribute__(attr)]
else:
attr_values = self.security_content_obj.__getattribute__(attr)
for attr_value in attr_values:
if attr_value == d_tags[d_tag]:
matched_deployments.append(d)
if len(matched_deployments) == 0:
self.security_content_obj.deployment = None
else:
self.security_content_obj.deployment = matched_deployments[-1]
def addRBA(self) -> None:
if self.security_content_obj:
risk_objects = []
risk_object_user_types = {'user', 'username', 'email address'}
risk_object_system_types = {'device', 'endpoint', 'hostname', 'ip address'}
@@ -71,20 +75,16 @@ class SecurityContentDetectionBuilder(DetectionBuilder):
for entity in self.security_content_obj.tags.observable:
risk_object = dict()
if entity['type'].lower() in risk_object_user_types:
for r in entity['role']:
if 'attacker' == r.lower() or 'victim' ==r.lower():
risk_object['risk_object_type'] = 'user'
risk_object['risk_object_field'] = entity['name']
risk_object['risk_score'] = self.security_content_obj.tags.risk_score
risk_objects.append(risk_object)
risk_object['risk_object_type'] = 'user'
risk_object['risk_object_field'] = entity['name']
risk_object['risk_score'] = self.security_content_obj.tags.risk_score
risk_objects.append(risk_object)
elif entity['type'].lower() in risk_object_system_types:
for r in entity['role']:
if 'attacker' == r.lower() or 'victim' ==r.lower():
risk_object['risk_object_type'] = 'system'
risk_object['risk_object_field'] = entity['name']
risk_object['risk_score'] = self.security_content_obj.tags.risk_score
risk_objects.append(risk_object)
risk_object['risk_object_type'] = 'system'
risk_object['risk_object_field'] = entity['name']
risk_object['risk_score'] = self.security_content_obj.tags.risk_score
risk_objects.append(risk_object)
else:
risk_object['threat_object_field'] = entity['name']
risk_object['threat_object_type'] = entity['type'].lower()
@@ -102,15 +102,13 @@ class SecurityContentDetectionBuilder(DetectionBuilder):
def addProvidingTechnologies(self) -> None:
if self.security_content_obj:
# if self.security_content_obj.tags.supported_tas:
if 'Endpoint' in self.security_content_obj.datamodel:
self.security_content_obj.providing_technologies = ["Sysmon", "Microsoft Windows","Carbon Black Response","CrowdStrike Falcon", "Symantec Endpoint Protection"]
if "`cloudtrail`" in str(self.security_content_obj.search):
self.security_content_obj.providing_technologies = ["Amazon Web Services - Cloudtrail"]
if '`wineventlog_security`' in self.security_content_obj.search or '`powershell`' in self.security_content_obj.search:
self.security_content_obj.providing_technologies = ["Microsoft Windows"]
if 'Endpoint' in str(self.security_content_obj.search):
self.security_content_obj.providing_technologies = ["Sysmon", "Microsoft Windows","Carbon Black Response","CrowdStrike Falcon", "Symantec Endpoint Protection"]
if "`cloudtrail`" in str(self.security_content_obj.search):
self.security_content_obj.providing_technologies = ["Amazon Web Services - Cloudtrail"]
if '`wineventlog_security`' in self.security_content_obj.search or '`powershell`' in self.security_content_obj.search:
self.security_content_obj.providing_technologies = ["Microsoft Windows"]
def addNesFields(self) -> None:
@@ -175,12 +173,11 @@ class SecurityContentDetectionBuilder(DetectionBuilder):
self.security_content_obj.baselines = matched_baselines
def addUnitTest(self, tests: list) -> None:
def addUnitTest(self) -> None:
if self.security_content_obj:
for test in tests:
if test.tests[0].name == self.security_content_obj.name:
self.security_content_obj.test = test
return
if self.security_content_obj.tests:
self.security_content_obj.test = self.security_content_obj.tests[0]
def addMitreAttackEnrichment(self, attack_enrichment: dict) -> None:
@@ -259,6 +256,55 @@ class SecurityContentDetectionBuilder(DetectionBuilder):
for splunk_app in self.security_content_obj.tags.supported_tas:
self.security_content_obj.splunk_app_enrichment.append(SplunkAppEnrichment.enrich_splunk_app(splunk_app, force_cached_or_offline=self.force_cached_or_offline))
def addCIS(self) -> None:
if self.security_content_obj:
if self.security_content_obj.tags.security_domain == "network":
self.security_content_obj.tags.cis20 = ["CIS 13"]
else:
self.security_content_obj.tags.cis20 = ["CIS 10"]
def addKillChainPhase(self) -> None:
if self.security_content_obj:
if not self.security_content_obj.tags.kill_chain_phases:
kill_chain_phases = list()
if self.security_content_obj.tags.mitre_attack_enrichments:
for mitre_attack_enrichment in self.security_content_obj.tags.mitre_attack_enrichments:
for mitre_attack_tactic in mitre_attack_enrichment.mitre_attack_tactics:
kill_chain_phases.append(ATTACK_TACTICS_KILLCHAIN_MAPPING[mitre_attack_tactic])
self.security_content_obj.tags.kill_chain_phases = list(dict.fromkeys(kill_chain_phases))
def addNist(self) -> None:
if self.security_content_obj:
if self.security_content_obj.type == "TTP":
self.security_content_obj.tags.nist = ["DE.CM"]
else:
self.security_content_obj.tags.nist = ["DE.AE"]
def addDatamodel(self) -> None:
if self.security_content_obj:
self.security_content_obj.datamodel = []
data_models = [
"Authentication",
"Change",
"Change_Analysis",
"Email",
"Endpoint",
"Network_Resolution",
"Network_Sessions",
"Network_Traffic",
"Risk",
"Splunk_Audit",
"UEBA",
"Updates",
"Vulnerabilities",
"Web"
]
for data_model in data_models:
if data_model in self.security_content_obj.search:
self.security_content_obj.datamodel.append(data_model)
def reset(self) -> None:
self.security_content_obj = None
bin/contentctl_project/contentctl_infrastructure/builder/security_content_director.py
+7 -3
View File
@@ -12,10 +12,15 @@ from bin.contentctl_project.contentctl_core.domain.entities.enums.enums import S
class SecurityContentDirector(Director):
def constructDetection(self, builder: DetectionBuilder, path: str, deployments: list, playbooks: list, baselines: list, tests: list, attack_enrichment: dict, macros: list, lookups: list, force_cached_or_offline: bool = False) -> None:
def constructDetection(self, builder: DetectionBuilder, path: str, deployments: list, playbooks: list, baselines: list, attack_enrichment: dict, macros: list, lookups: list, force_cached_or_offline: bool = False) -> None:
builder.reset()
builder.setObject(os.path.join(os.path.dirname(__file__), path))
builder.addDeployment(deployments)
builder.addMitreAttackEnrichment(attack_enrichment)
builder.addKillChainPhase()
builder.addCIS()
builder.addNist()
builder.addDatamodel()
builder.addRBA()
builder.addProvidingTechnologies()
builder.addNesFields()
@@ -23,8 +28,7 @@ class SecurityContentDirector(Director):
builder.addMappings()
builder.addBaseline(baselines)
builder.addPlaybook(playbooks)
builder.addUnitTest(tests)
builder.addMitreAttackEnrichment(attack_enrichment)
builder.addUnitTest()
builder.addMacros(macros)
builder.addLookups(lookups)
builder.addCve()
bin/contentctl_project/contentctl_infrastructure/builder/security_content_story_builder.py
+1
View File
@@ -95,6 +95,7 @@ class SecurityContentStoryBuilder(StoryBuilder):
"tags": tags_obj
})
datamodels.update(detection.datamodel)
if detection.tags.kill_chain_phases:
kill_chain_phases.update(detection.tags.kill_chain_phases)
bin/contentctl_project/contentctl_infrastructure/builder/sigma_converter.py
+408
View File
@@ -0,0 +1,408 @@
import os
import sys
import copy
from dataclasses import dataclass
from jinja2 import Environment, FileSystemLoader
from sigma.processing.conditions import LogsourceCondition
from sigma.processing.transformations import AddConditionTransformation, FieldMappingTransformation, DetectionItemFailureTransformation, RuleFailureTransformation, SetStateTransformation
from sigma.processing.conditions import LogsourceCondition, IncludeFieldCondition, ExcludeFieldCondition, RuleProcessingItemAppliedCondition
from sigma.collection import SigmaCollection
from sigma.backends.splunk import SplunkBackend
from sigma.processing.pipeline import ProcessingItem, ProcessingPipeline
from bin.contentctl_project.contentctl_infrastructure.builder.utils import Utils
from bin.contentctl_project.contentctl_core.domain.entities.enums.enums import SigmaConverterTarget
from bin.contentctl_project.contentctl_infrastructure.builder.yml_reader import YmlReader
from bin.contentctl_project.contentctl_core.domain.entities.detection import Detection
from bin.contentctl_project.contentctl_core.domain.entities.data_source import DataSource
from bin.contentctl_project.contentctl_infrastructure.builder.backend_splunk_ba import SplunkBABackend
from bin.contentctl_project.contentctl_core.application.factory.utils.utils import Utils
from bin.contentctl_project.contentctl_core.domain.constants.constants import *
@dataclass(frozen=True)
class SigmaConverterInputDto:
data_model: SigmaConverterTarget
detection_path: str
detection_folder : str
input_path: str
log_source: str
cim_to_ocsf: bool
@dataclass(frozen=True)
class SigmaConverterOutputDto:
detections: list
class SigmaConverter():
output_dto : SigmaConverterOutputDto
def __init__(self, output_dto: SigmaConverterOutputDto) -> None:
self.output_dto = output_dto
def execute(self, input_dto: SigmaConverterInputDto) -> None:
detection_files = []
errors = []
if input_dto.detection_path:
detection_files.append(input_dto.detection_path)
elif input_dto.detection_folder:
detection_files = Utils.get_all_yml_files_from_directory(input_dto.detection_folder)
else:
print("ERROR: --detection_path or --detection_folder needed.")
sys.exit(1)
for detection_file in detection_files:
#try:
detection = self.read_detection(str(detection_file))
print("Converting detection: " + detection.name)
data_source = self.load_data_source(input_dto.input_path, detection.data_source[0])
if not data_source:
print("ERROR: Didn't find data source with name: " + detection.data_source[0] + " for detection " + detection.name)
sys.exit(1)
file_name = detection.name.replace(' ', '_').replace('-','_').replace('.','_').replace('/','_').lower()
if input_dto.data_model == SigmaConverterTarget.RAW:
if input_dto.log_source and input_dto.log_source != detection.data_source[0][0]:
try:
field_mapping = self.find_mapping(data_source.convert_to_log_source, 'data_source', input_dto.log_source)
except Exception as e:
print(e)
print("ERROR: Couldn't find data source mapping for log source " + input_dto.log_source + " for detection: " + detection.name)
sys.exit(1)
detection = self.convert_detection_fields(detection, field_mapping)
logsource_condition = self.get_logsource_condition(data_source)
processing_item = self.get_field_transformation_processing_item(
field_mapping['mapping'],
logsource_condition
)
sigma_processing_pipeline = self.get_pipeline_from_processing_items([processing_item])
splunk_backend = SplunkBackend(processing_pipeline=sigma_processing_pipeline)
data_source = self.load_data_source(input_dto.input_path, input_dto.log_source)
else:
splunk_backend = SplunkBackend()
sigma_rule = self.get_sigma_rule(detection, data_source)
search = splunk_backend.convert(sigma_rule)[0]
search = self.add_source_macro(search, data_source.type)
search = self.add_stats_count(search, data_source.raw_fields)
search = self.add_timeformat_conversion(search)
search = self.add_filter_macro(search, file_name)
detection.file_path = file_name + '.yml'
elif input_dto.data_model == SigmaConverterTarget.CIM:
logsource_condition = self.get_logsource_condition(data_source)
try:
field_mapping = self.find_mapping(data_source.field_mappings, 'data_model', 'cim')
except Exception as e:
print(e)
print("ERROR: Couldn't find data source mapping to cim for log source " + detection.data_source[0] + " and detection " + detection.name)
sys.exit(1)
detection = self.convert_detection_fields(detection, field_mapping)
sigma_rule = self.get_sigma_rule(detection, data_source)
sigma_transformation_processing_item = self.get_field_transformation_processing_item(
field_mapping['mapping'],
logsource_condition
)
sigma_state_fields_processing_item = self.get_state_fields_processing_item(
field_mapping['mapping'].values(),
logsource_condition
)
sigma_state_data_model_processing_item = self.get_state_data_model_processing_item(
field_mapping['data_set'],
logsource_condition
)
sigma_processing_pipeline = self.get_pipeline_from_processing_items([
sigma_transformation_processing_item,
sigma_state_fields_processing_item,
sigma_state_data_model_processing_item
])
splunk_backend = SplunkBackend(processing_pipeline=sigma_processing_pipeline)
search = splunk_backend.convert(sigma_rule, "data_model")[0]
search = self.add_filter_macro(search, file_name)
detection.file_path = file_name + '.yml'
elif input_dto.data_model == SigmaConverterTarget.OCSF:
processing_items = list()
logsource_condition = self.get_logsource_condition(data_source)
if input_dto.log_source and input_dto.log_source != detection.data_source[0]:
data_source_new = self.load_data_source(input_dto.input_path, input_dto.log_source)
try:
field_mapping = self.get_mapping_converted_data_source(
data_source,
"data_source",
input_dto.log_source,
data_source_new,
"data_model",
"ocsf"
)
except Exception as e:
print(e)
print("ERROR: Couldn't find data source mapping for log source " + input_dto.log_source + " and detection " + detection.name)
sys.exit(1)
cim_to_ocsf_mapping = self.get_cim_to_ocsf_mapping(data_source_new)
elif input_dto.cim_to_ocsf:
field_mapping = self.get_cim_to_ocsf_mapping(data_source)
cim_to_ocsf_mapping = field_mapping
else:
field_mapping = self.find_mapping(data_source.field_mappings, 'data_model', 'ocsf')
cim_to_ocsf_mapping = self.get_cim_to_ocsf_mapping(data_source)
field_mapping_underline = copy.deepcopy(field_mapping)
for field in field_mapping_underline["mapping"].keys():
field_mapping_underline["mapping"][field] = field_mapping_underline["mapping"][field].replace(".", "_")
self.add_required_fields(field_mapping, detection)
self.add_mappings(cim_to_ocsf_mapping, detection)
self.update_observables(detection)
processing_items.append(
self.get_field_transformation_processing_item(
field_mapping_underline['mapping'],
logsource_condition
)
)
processing_items.append(
self.get_state_fields_processing_item(
field_mapping_underline['mapping'].values(),
logsource_condition
)
)
detection = self.convert_detection_fields(detection, field_mapping_underline)
sigma_rule = self.get_sigma_rule(detection, data_source)
sigma_processing_pipeline = self.get_pipeline_from_processing_items(processing_items)
splunk_backend = SplunkBABackend(processing_pipeline=sigma_processing_pipeline, detection=detection, field_mapping=field_mapping)
search = splunk_backend.convert(sigma_rule, "data_model")[0]
search = search + ' --finding_report--'
detection.file_path = 'ssa___' + file_name + '.yml'
detection.search = search
self.output_dto.detections.append(detection)
# except Exception as e:
# print(e)
# errors.append("ERROR: Converting detection " + detection.name)
print()
for error in errors:
print(error)
print()
def read_detection(self, detection_path : str) -> Detection:
yml_dict = YmlReader.load_file(detection_path)
yml_dict["tags"]["name"] = yml_dict["name"]
detection = Detection.parse_obj(yml_dict)
detection.source = os.path.split(os.path.dirname(detection_path))[-1]
return detection
def load_data_source(self, input_path: str, data_source_name: str) -> DataSource:
data_sources = list()
files = Utils.get_all_yml_files_from_directory(os.path.join(input_path, 'data_sources'))
for file in files:
data_sources.append(DataSource.parse_obj(YmlReader.load_file(str(file))))
data_source = None
for obj in data_sources:
if obj.name == data_source_name:
return obj
return None
def get_sigma_rule(self, detection: Detection, data_source: DataSource) -> SigmaCollection:
return SigmaCollection.from_dicts([{
"title": detection.name,
"status": "experimental",
"logsource": {
"category": data_source.category,
"product": data_source.product
},
"detection": detection.search
}])
def convert_detection_fields(self, detection: Detection, mappings: dict) -> Detection:
for selection in detection.search.keys():
if selection != "condition":
new_selection = copy.deepcopy(detection.search[selection])
for field in detection.search[selection].keys():
for mapping in mappings["mapping"].keys():
if mapping == field:
new_selection[mappings["mapping"][mapping]] = detection.search[selection][field]
new_selection.pop(field)
detection.search[selection] = new_selection
return detection
def get_logsource_condition(self, data_source: DataSource) -> LogsourceCondition:
return LogsourceCondition(
category=data_source.category,
product=data_source.product,
)
def get_field_transformation_processing_item(self, data_source_mapping: dict, logsource_condition: LogsourceCondition) -> ProcessingItem:
return ProcessingItem(
identifier="field_mapping_transformation",
transformation=FieldMappingTransformation(data_source_mapping),
rule_conditions=[
logsource_condition
]
)
def get_state_fields_processing_item(self, fields: list, logsource_condition: LogsourceCondition) -> ProcessingItem:
return ProcessingItem(
identifier="fields",
transformation=SetStateTransformation("fields", fields),
rule_conditions=[
logsource_condition
]
)
def get_state_data_model_processing_item(self, data_model: str, logsource_condition: LogsourceCondition) -> ProcessingItem:
return ProcessingItem(
identifier="data_model",
transformation=SetStateTransformation("data_model_set", data_model),
rule_conditions=[
logsource_condition
]
)
def get_pipeline_from_processing_items(self, processing_items: list) -> ProcessingPipeline:
return ProcessingPipeline(
name="Splunk Sigma",
priority=10,
items=processing_items
)
def add_source_macro(self, search: str, data_source_type: str) -> str:
return "`" + data_source_type + "` " + search
def add_stats_count(self, search: str, fields: list) -> str:
search = search + " | fillnull | stats count min(_time) as firstTime max(_time) as lastTime by "
for key in fields:
search = search + key + " "
return search
def add_timeformat_conversion(self, search: str) -> str:
return search + '| convert timeformat="%Y-%m-%dT%H:%M:%S" ctime(firstTime) | convert timeformat="%Y-%m-%dT%H:%M:%S" ctime(lastTime) '
def add_filter_macro(self, search: str, file_name: str) -> str:
return search + '| `' + file_name + '_filter`'
def find(self, name: str, path: str) -> str:
for root, dirs, files in os.walk(path):
if name in files:
return os.path.join(root, name)
return None
def find_mapping(self, field_mappings: list, object: str, data_model: str) -> dict:
for mapping in field_mappings:
if mapping[object] == data_model:
return mapping
raise AttributeError("ERROR: Couldn't find mapping.")
def add_required_fields(self, field_mapping: dict, detection: Detection) -> None:
required_fields = list()
required_fields = ["process.user.name", "device.hostname"]
for mapping in field_mapping["mapping"].keys():
for selection in detection.search.keys():
if selection != "condition":
for detection_field in detection.search[selection]:
if detection_field.startswith(mapping):
if not field_mapping["mapping"][mapping] in required_fields:
required_fields.append(field_mapping["mapping"][mapping])
detection.tags.required_fields = required_fields
def add_mappings(self, field_mapping: dict, detection: Detection) -> None:
mappings = list()
for mapping in field_mapping["mapping"].keys():
mappings.append({
"ocsf": field_mapping["mapping"][mapping],
"cim": mapping
})
detection.tags.mappings = mappings
def update_observables(self, detection : Detection) -> None:
mapping_field_to_type = {
"process.user.name": "User Name",
"device.hostname": "Hostname",
"process.file.name": "File Name",
"actor.process.file.name": "File Name",
"actor.process.file.path": "File Name",
"actor.process.cmd_line": "Process",
"process.cmd_line": "Other",
"process.file.path": "File",
"process.file.name": "File"
}
observables = list()
for field in detection.tags.required_fields:
observables.append({
"name": field,
"type": mapping_field_to_type[field]
})
detection.tags.observable = observables
def get_cim_to_ocsf_mapping(self, data_source : DataSource) -> dict:
cim_to_ocsf_mapping = dict()
cim_to_ocsf_mapping["mapping"] = dict()
cim_mapping = self.find_mapping(data_source.field_mappings, "data_model", "cim")
ocsf_mapping = self.find_mapping(data_source.field_mappings, "data_model", "ocsf")
for key in cim_mapping["mapping"].keys():
cim_field = cim_mapping["mapping"][key].split(".")[1]
cim_to_ocsf_mapping["mapping"][cim_field] = ocsf_mapping["mapping"][key]
return cim_to_ocsf_mapping
def get_mapping_converted_data_source(self, det_ds: DataSource, det_ds_obj: str, det_ds_dm: str, con_ds: DataSource, con_ds_obj: str, con_ds_dm: str) -> dict:
mapping = dict()
mapping["mapping"] = dict()
det_ds_mapping = self.find_mapping(det_ds.convert_to_log_source, det_ds_obj, det_ds_dm)
con_ds_mapping = self.find_mapping(con_ds.field_mappings, con_ds_obj, con_ds_dm)
for key in det_ds_mapping["mapping"].keys():
mapped_field = con_ds_mapping["mapping"][det_ds_mapping["mapping"][key]]
mapping["mapping"][key] = mapped_field
return mapping
bin/contentctl_project/contentctl_infrastructure/builder/utils.py
+13
View File
@@ -0,0 +1,13 @@
import os
class Utils:
@staticmethod
def get_all_yml_files_from_directory(path: str) -> list:
listOfFiles = list()
for (dirpath, dirnames, filenames) in os.walk(path):
for file in filenames:
if file.endswith(".yml"):
listOfFiles.append(os.path.join(dirpath, file))
return sorted(listOfFiles)

Some files were not shown because too many files have changed in this diff Show More