admin/splunk-security_content
1
0
Fork 0
mirror of https://github.com/splunk/security_content synced 2026-06-08 17:32:49 +00:00
Code Issues Packages Projects Releases Wiki Activity

Branch was auto-updated.

Browse Source
This commit is contained in:
github-actions[bot]
2021-04-02 18:24:54 +00:00
committed by GitHub
parent 96096effb7 fff8443bba
commit 0809fa39b2
158 changed files with 137771 additions and 28 deletions
Download Patch File Download Diff File Expand all files Collapse all files
detections/endpoint/any_powershell_downloadfile.yml
+1
View File
@@ -31,6 +31,7 @@ references:
tags:
analytic_story:
- Malicious PowerShell
- Ingress Tool Transfer
dataset:
- https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1059.001/atomic_red_team/windows-sysmon.log
kill_chain_phases:
detections/endpoint/any_powershell_downloadstring.yml
+1
View File
@@ -32,6 +32,7 @@ tags:
analytic_story:
- Malicious PowerShell
- HAFNIUM Group
- Ingress Tool Transfer
automated_detection_testing: passed
dataset:
- https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1059.001/atomic_red_team/windows-sysmon.log
detections/endpoint/bits_job_persistence.yml
+53
View File
@@ -0,0 +1,53 @@
name: BITS Job Persistence
id: e97a5ffe-90bf-11eb-928a-acde48001122
version: 1
date: '2021-03-29'
author: Michael Haag, Splunk
type: batch
datamodel:
- Endpoint
description: The following query identifies Microsoft Background Intelligent Transfer
Service utility `bitsadmin.exe` scheduling a BITS job to persist on an endpoint.
The query identifies the parameters used to create, resume or add a file to a BITS
job. Typically seen combined in a oneliner or ran in sequence. If identified, review the BITS job created and capture any files written to disk. It is possible for BITS to be used to upload files and this may require further network data analysis to identify. You can use `bitsadmin /list /verbose`
to list out the jobs during investigation.
search: '| tstats `security_content_summariesonly` count min(_time) as firstTime max(_time)
as lastTime from datamodel=Endpoint.Processes where Processes.process_name=bitsadmin.exe
Processes.process IN (*create*, *addfile*, *setnotifyflags*, *setnotifycmdline*,
*setminretrydelay*, *setcustomheaders*, *resume* ) by Processes.dest Processes.user
Processes.parent_process Processes.process_name Processes.process Processes.process_id
Processes.parent_process_id | `drop_dm_object_name(Processes)` | `security_content_ctime(firstTime)`
| `security_content_ctime(lastTime)` | `bits_job_persistence_filter`'
how_to_implement: To successfully implement this search you need to be ingesting information
on process that include the name of the process responsible for the changes from
your endpoints into the `Endpoint` datamodel in the `Processes` node.
known_false_positives: Limited false positives will be present. Typically, applications
will use `BitsAdmin.exe`. Any filtering should be done based on command-line arguments
(legitimate applications) or parent process.
references:
- https://attack.mitre.org/techniques/T1197/
- https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/bitsadmin
- https://github.com/redcanaryco/atomic-red-team/blob/master/atomics/T1197/T1197.md#atomic-test-3---persist-download--execute
- https://lolbas-project.github.io/lolbas/Binaries/Bitsadmin/
tags:
analytic_story:
- BITS Jobs
dataset:
- https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1197/atomic_red_team/windows-sysmon.log
kill_chain_phases:
- Exploitation
mitre_attack_id:
- T1197
product:
- Splunk Enterprise
- Splunk Enterprise Security
- Splunk Cloud
required_fields:
- _time
- Processes.process
- Processes.parent_process
- Processes.process_name
- Processes.user
- Processes.dest
security_domain: endpoint
automated_detection_testing: passed
detections/endpoint/bitsadmin_download_file.yml
+60
View File
@@ -0,0 +1,60 @@
name: BITSAdmin Download File
id: 80630ff4-8e4c-11eb-aab5-acde48001122
version: 1
date: '2021-03-26'
author: Michael Haag, Splunk
type: batch
datamodel:
- Endpoint
description: The following query identifies Microsoft Background Intelligent Transfer
Service utility `bitsadmin.exe` using the `transfer` parameter to download a remote
object. In addition, look for `download` or `upload` on the command-line, the switches
are not required to perform a transfer. Capture any files downloaded. Review the
reputation of the IP or domain used. Typically once executed, a follow on command
will be used to execute the dropped file. Note that the network connection or file
modification events related will not spawn or create from `bitsadmin.exe`, but the
artifacts will appear in a parallel process of `svchost.exe` with a command-line
similar to `svchost.exe -k netsvcs -s BITS`. It's important to review all parallel
and child processes to capture any behaviors and artifacts. In some suspicious and
malicious instances, BITS jobs will be created. You can use `bitsadmin /list /verbose`
to list out the jobs during investigation.
search: '| tstats `security_content_summariesonly` count min(_time) as firstTime max(_time)
as lastTime from datamodel=Endpoint.Processes where Processes.process_name=bitsadmin.exe
Processes.process=*transfer* by Processes.dest Processes.user Processes.parent_process
Processes.process_name Processes.process Processes.process_id Processes.parent_process_id
| `drop_dm_object_name(Processes)` | `security_content_ctime(firstTime)` | `security_content_ctime(lastTime)`
| `bitsadmin_download_file_filter`'
how_to_implement: To successfully implement this search you need to be ingesting information
on process that include the name of the process responsible for the changes from
your endpoints into the `Endpoint` datamodel in the `Processes` node.
known_false_positives: Limited false positives, however it may be required to filter
based on parent process name or network connection.
references:
- https://github.com/redcanaryco/atomic-red-team/blob/8eb52117b748d378325f7719554a896e37bccec7/atomics/T1105/T1105.md#atomic-test-9---windows---bitsadmin-bits-download
- https://github.com/redcanaryco/atomic-red-team/blob/bc705cb7aaa5f26f2d96585fac8e4c7052df0ff9/atomics/T1197/T1197.md
- https://docs.microsoft.com/en-us/windows/win32/bits/bitsadmin-tool
- https://thedfirreport.com/2021/03/29/sodinokibi-aka-revil-ransomware/
tags:
analytic_story:
- Ingress Tool Transfer
- BITS Jobs
dataset:
- https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1197/atomic_red_team/windows-sysmon.log
kill_chain_phases:
- Exploitation
mitre_attack_id:
- T1197
- T1105
product:
- Splunk Enterprise
- Splunk Enterprise Security
- Splunk Cloud
required_fields:
- _time
- Processes.process
- Processes.parent_process
- Processes.process_name
- Processes.user
- Processes.dest
security_domain: endpoint
automated_detection_testing: passed
detections/endpoint/dsquery_domain_discovery.yml
+66
View File
@@ -0,0 +1,66 @@
name: DSQuery Domain Discovery
id: cc316032-924a-11eb-91a2-acde48001122
version: 1
date: '2021-03-31'
author: Michael Haag, Splunk
type: batch
datamodel:
- Endpoint
description: 'The following analytic identifies "dsquery.exe" execution with arguments
looking for `TrustedDomain` query directly on the command-line. This is typically
indicative of an Administrator or adversary perform domain trust discovery. Note
that this query does not identify any other variations of "Dsquery.exe" usage.\
Within this detection, it is assumed `dsquery.exe` is not moved or renamed.\
The search will return the first time and last time these command-line arguments
were used for these executions, as well as the target system, the user, process
"dsquery.exe" and its parent process.\
DSQuery.exe is natively found in `C:\Windows\system32` and `C:\Windows\syswow64`
and only on Server operating system.\
The following DLL(s) are loaded when DSQuery.exe is launched `dsquery.dll`. If found
loaded by another process, it is possible dsquery is running within that process
context in memory.\
In addition to trust discovery, review parallel processes for additional behaviors
performed. Identify the parent process and capture any files (batch files, for example)
being used.'
search: '| tstats `security_content_summariesonly` count min(_time) as firstTime max(_time)
as lastTime from datamodel=Endpoint.Processes where Processes.process_name=dsquery.exe
Processes.process=*trustedDomain* by Processes.dest Processes.user Processes.parent_process
Processes.process_name Processes.process Processes.process_id Processes.parent_process_id
| `drop_dm_object_name(Processes)` | `security_content_ctime(firstTime)` | `security_content_ctime(lastTime)`
| `dsquery_domain_discovery_filter`'
how_to_implement: To successfully implement this search you need to be ingesting information
on process that include the name of the process responsible for the changes from
your endpoints into the `Endpoint` datamodel in the `Processes` node.
known_false_positives: Limited false positives. If there is a true false positive,
filter based on command-line or parent process.
references:
- https://github.com/redcanaryco/atomic-red-team/blob/master/atomics/T1482/T1482.md
- http://www.harmj0y.net/blog/redteaming/a-guide-to-attacking-domain-trusts/
- https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2012-R2-and-2012/cc732952(v=ws.11)
- https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2012-R2-and-2012/cc754232(v=ws.11)
tags:
analytic_story:
- Domain Trust Discovery
dataset:
- https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1482/atomic_red_team/windows-sysmon.log
kill_chain_phases:
- Exploitation
mitre_attack_id:
- T1482
product:
- Splunk Enterprise
- Splunk Enterprise Security
- Splunk Cloud
required_fields:
- _time
- Processes.process_name
- Processes.process
- Processes.user
- Processes.dest
security_domain: endpoint
automated_detection_testing: passed
detections/endpoint/nltest_domain_trust_discovery.yml
+1
View File
@@ -33,6 +33,7 @@ references:
tags:
analytic_story:
- Ryuk Ransomware
- Domain Trust Discovery
asset_type: Endpoint
automated_detection_testing: passed
cis20:
detections/endpoint/powershell_start_bitstransfer.yml
+52
View File
@@ -0,0 +1,52 @@
name: PowerShell Start-BitsTransfer
id: 39e2605a-90d8-11eb-899e-acde48001122
version: 1
date: '2021-03-29'
author: Michael Haag, Splunk
type: batch
datamodel:
- Endpoint
description: Start-BitsTransfer is the PowerShell "version" of BitsAdmin.exe. Similar
functionality is present. This technique variation is not as commonly used by adversaries,
but has been abused in the past. Lesser known uses include the ability to set the
`-TransferType` to `Upload` for exfiltration of files. In an instance where `Upload`
is used, it is highly possible files will be archived. During triage, review parallel
processes and process lineage. Capture any files on disk and review. For the remote
domain or IP, what is the reputation?
search: '| tstats `security_content_summariesonly` count min(_time) as firstTime max(_time)
as lastTime from datamodel=Endpoint.Processes where Processes.process_name=powershell.exe
Processes.process=*start-bitstransfer* by Processes.dest Processes.user Processes.parent_process
Processes.process_name Processes.process Processes.process_id Processes.parent_process_id
| `drop_dm_object_name(Processes)` | `security_content_ctime(firstTime)` | `security_content_ctime(lastTime)`
| `powershell_start_bitstransfer_filter`'
how_to_implement: To successfully implement this search you need to be ingesting information
on process that include the name of the process responsible for the changes from
your endpoints into the `Endpoint` datamodel in the `Processes` node.
known_false_positives: Limited false positives. It is possible administrators will
utilize Start-BitsTransfer for administrative tasks, otherwise filter based parent
process or command-line arguments.
references:
- https://isc.sans.edu/diary/Investigating+Microsoft+BITS+Activity/23281
- https://docs.microsoft.com/en-us/windows/win32/bits/using-windows-powershell-to-create-bits-transfer-jobs
tags:
analytic_story:
- BITS Jobs
dataset:
- https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1197/atomic_red_team/windows-sysmon.log
kill_chain_phases:
- Exploitation
mitre_attack_id:
- T1197
product:
- Splunk Enterprise
- Splunk Enterprise Security
- Splunk Cloud
required_fields:
- _time
- Processes.process
- Processes.parent_process
- Processes.process_name
- Processes.user
- Processes.dest
security_domain: endpoint
automated_detection_testing: passed
detections/experimental/endpoint/windows_adfind_exe.yml
+1
View File
@@ -30,6 +30,7 @@ references:
tags:
analytic_story:
- NOBELIUM Group
- Domain Trust Discovery
asset_type: Endpoint
cis20:
- CIS 8
docs/detections.md
+1 -1
View File
@@ -8190,7 +8190,7 @@ The following analytics are designed to identifies some CLOP ransomware variant
#### Search
```
| tstats `security_content_summariesonly` values(Processes.process) as cmdline values(Processes.parent_process_name) as parent_process values(Processes.process_name) count min(_time) as firstTime max(_time) as lastTime from datamodel=Endpoint.Processes where Processes.process = "*runrun*" OR Processes.process = "*temp.dat*" by Processes.parent_process_name Processes.process_name Processes.process Processes.dest Processes.user Processes.process_id Processes.process_guid
| tstats `security_content_summariesonly` values(Processes.process) as cmdline values(Processes.parent_process_name) as parent_process values(Processes.process_name) count min(_time) as firstTime max(_time) as lastTime from datamodel=Endpoint.Processes where Processes.process_name != "*temp.dat*" Processes.process = "*runrun*" OR Processes.process = "*temp.dat*" by Processes.parent_process_name Processes.process_name Processes.process Processes.dest Processes.user Processes.process_id Processes.process_guid
| `drop_dm_object_name(Processes)`
| `security_content_ctime(firstTime)`
| `security_content_ctime(lastTime)`
docs/detections.wiki
+2 -2
View File
@@ -13106,7 +13106,7 @@ The following analytics are designed to identifies some CLOP ransomware variant
====Search====
<search>
| tstats `security_content_summariesonly` values(Processes.process) as cmdline values(Processes.parent_process_name) as parent_process values(Processes.process_name) count min(_time) as firstTime max(_time) as lastTime from datamodel=Endpoint.Processes where Processes.process = "*runrun*" OR Processes.process = "*temp.dat*" by Processes.parent_process_name Processes.process_name Processes.process Processes.dest Processes.user Processes.process_id Processes.process_guid
| tstats `security_content_summariesonly` values(Processes.process) as cmdline values(Processes.parent_process_name) as parent_process values(Processes.process_name) count min(_time) as firstTime max(_time) as lastTime from datamodel=Endpoint.Processes where Processes.process_name != "*temp.dat*" Processes.process = "*runrun*" OR Processes.process = "*temp.dat*" by Processes.parent_process_name Processes.process_name Processes.process Processes.dest Processes.user Processes.process_id Processes.process_guid
| `drop_dm_object_name(Processes)`
| `security_content_ctime(firstTime)`
| `security_content_ctime(lastTime)`
@@ -31671,7 +31671,7 @@ There might be false positives associted with this detection since items like ar
''#############''
''# Automatically generated by doc_gen.py in https://github.com/splunk/security_content''
''# On Date: 2021-03-29 18:42:22.264486 UTC''
''# On Date: 2021-04-02 17:10:21.330169 UTC''
''# Author: Splunk Security Research''
''# Contact: research@splunk.com''
''#############''
docs/stories.md
+12 -12
View File
@@ -3096,7 +3096,7 @@ _version_: 1
### AWS Network ACL Activity
Monitor your AWS network infrastructure for bad configurations and malicious activity. Investigative searches help you probe deeper, when the facts warrant it.
- **Product**: Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud
- **Product**: Splunk Security Analytics for AWS, Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud
- **Datamodel**:
- **ATT&CK**: [T1562.007](https://attack.mitre.org/techniques/T1562.007/)
- **Last Updated**: 2018-05-21
@@ -3143,7 +3143,7 @@ _version_: 2
### AWS Security Hub Alerts
This story is focused around detecting Security Hub alerts generated from AWS
- **Product**: Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud
- **Product**: Splunk Security Analytics for AWS, Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud
- **Datamodel**:
- **ATT&CK**:
- **Last Updated**: 2020-08-04
@@ -3267,7 +3267,7 @@ _version_: 1
### Cloud Cryptomining
Monitor your cloud compute instances for activities related to cryptojacking/cryptomining. New instances that originate from previously unseen regions, users who launch abnormally high numbers of instances, or compute instances started by previously unseen users are just a few examples of potentially malicious behavior.
- **Product**: Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud
- **Product**: Splunk Security Analytics for AWS, Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud
- **Datamodel**: Change
- **ATT&CK**: [T1078.004](https://attack.mitre.org/techniques/T1078.004/), [T1535](https://attack.mitre.org/techniques/T1535/)
- **Last Updated**: 2019-10-02
@@ -3313,7 +3313,7 @@ _version_: 1
### Cloud Federated Credential Abuse
This analytical story addresses events that indicate abuse of cloud federated credentials. These credentials are usually extracted from endpoint desktop or servers specially those servers that provide federation services such as Windows Active Directory Federation Services. Identity Federation relies on objects such as Oauth2 tokens, cookies or SAML assertions in order to provide seamless access between cloud and perimeter environments. If these objects are either hijacked or forged then attackers will be able to pivot into victim's cloud environements.
- **Product**: Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud
- **Product**: Splunk Security Analytics for AWS, Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud
- **Datamodel**: Endpoint
- **ATT&CK**: [T1003.001](https://attack.mitre.org/techniques/T1003.001/), [T1078](https://attack.mitre.org/techniques/T1078/), [T1136.003](https://attack.mitre.org/techniques/T1136.003/), [T1546.012](https://attack.mitre.org/techniques/T1546.012/), [T1556](https://attack.mitre.org/techniques/T1556/)
- **Last Updated**: 2021-01-26
@@ -3615,7 +3615,7 @@ _version_: 1
### Office 365 Detections
This story is focused around detecting Office 365 Attacks.
- **Product**: Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud
- **Product**: Splunk Security Analytics for AWS, Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud
- **Datamodel**:
- **ATT&CK**: [T1110](https://attack.mitre.org/techniques/T1110/), [T1110.001](https://attack.mitre.org/techniques/T1110.001/), [T1114](https://attack.mitre.org/techniques/T1114/), [T1114.002](https://attack.mitre.org/techniques/T1114.002/), [T1114.003](https://attack.mitre.org/techniques/T1114.003/), [T1136.003](https://attack.mitre.org/techniques/T1136.003/), [T1556](https://attack.mitre.org/techniques/T1556/), [T1562.007](https://attack.mitre.org/techniques/T1562.007/)
- **Last Updated**: 2020-12-16
@@ -3733,7 +3733,7 @@ _version_: 1
### Suspicious AWS Login Activities
Monitor your AWS authentication events using your CloudTrail logs. Searches within this Analytic Story will help you stay aware of and investigate suspicious logins.
- **Product**: Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud
- **Product**: Splunk Security Analytics for AWS, Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud
- **Datamodel**: Authentication
- **ATT&CK**: [T1078.004](https://attack.mitre.org/techniques/T1078.004/), [T1535](https://attack.mitre.org/techniques/T1535/)
- **Last Updated**: 2019-05-01
@@ -3777,7 +3777,7 @@ _version_: 1
### Suspicious AWS S3 Activities
Use the searches in this Analytic Story to monitor your AWS S3 buckets for evidence of anomalous activity and suspicious behaviors, such as detecting open S3 buckets and buckets being accessed from a new IP. The contextual and investigative searches will give you more information, when required.
- **Product**: Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud
- **Product**: Splunk Security Analytics for AWS, Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud
- **Datamodel**:
- **ATT&CK**: [T1530](https://attack.mitre.org/techniques/T1530/)
- **Last Updated**: 2018-07-24
@@ -3860,7 +3860,7 @@ _version_: 1
### Suspicious Cloud Authentication Activities
Monitor your cloud authentication events. Searches within this Analytic Story leverage the recent cloud updates to the Authentication data model to help you stay aware of and investigate suspicious login activity.
- **Product**: Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud
- **Product**: Splunk Security Analytics for AWS, Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud
- **Datamodel**: Authentication
- **ATT&CK**: [T1535](https://attack.mitre.org/techniques/T1535/)
- **Last Updated**: 2020-06-04
@@ -3908,7 +3908,7 @@ _version_: 1
### Suspicious Cloud Instance Activities
Monitor your cloud infrastructure provisioning activities for behaviors originating from unfamiliar or unusual locations. These behaviors may indicate that malicious activities are occurring somewhere within your cloud environment.
- **Product**: Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud
- **Product**: Splunk Security Analytics for AWS, Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud
- **Datamodel**: Change
- **ATT&CK**: [T1078.004](https://attack.mitre.org/techniques/T1078.004/)
- **Last Updated**: 2020-08-25
@@ -3949,7 +3949,7 @@ _version_: 1
### Suspicious Cloud Provisioning Activities
Monitor your cloud infrastructure provisioning activities for behaviors originating from unfamiliar or unusual locations. These behaviors may indicate that malicious activities are occurring somewhere within your cloud environment.
- **Product**: Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud
- **Product**: Splunk Security Analytics for AWS, Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud
- **Datamodel**: Change
- **ATT&CK**: [T1078](https://attack.mitre.org/techniques/T1078/)
- **Last Updated**: 2018-08-20
@@ -3990,7 +3990,7 @@ _version_: 1
### Suspicious Cloud User Activities
Detect and investigate suspicious activities by users and roles in your cloud environments.
- **Product**: Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud
- **Product**: Splunk Security Analytics for AWS, Splunk Security Analytics for AWS, Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud
- **Datamodel**: Change
- **ATT&CK**: [T1078](https://attack.mitre.org/techniques/T1078/), [T1078.004](https://attack.mitre.org/techniques/T1078.004/)
- **Last Updated**: 2020-09-04
@@ -4658,7 +4658,7 @@ _version_: 1
### Ransomware Cloud
Leverage searches that allow you to detect and investigate unusual activities that might relate to ransomware. These searches include cloud related objects that may be targeted by malicious actors via cloud providers own encryption features.
- **Product**: Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud
- **Product**: Splunk Security Analytics for AWS, Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud
- **Datamodel**:
- **ATT&CK**: [T1486](https://attack.mitre.org/techniques/T1486/)
- **Last Updated**: 2020-10-27
docs/stories.wiki
+13 -13
View File
@@ -4081,7 +4081,7 @@ This analytic story contains detections that query your AWS Cloudtrail for activ
===Aws network acl activity===
Monitor your AWS network infrastructure for bad configurations and malicious activity. Investigative searches help you probe deeper, when the facts warrant it.
* '''Product''': Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud
* '''Product''': Splunk Security Analytics for AWS, Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud
* '''Datamodel''':
* '''ATT&CK''': [https://attack.mitre.org/techniques/T1562.007/ T1562.007]
* '''Last Updated''': 2018-05-21
@@ -4136,7 +4136,7 @@ Monitor your AWS network infrastructure for bad configurations and malicious act
===Aws security hub alerts===
This story is focused around detecting Security Hub alerts generated from AWS
* '''Product''': Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud
* '''Product''': Splunk Security Analytics for AWS, Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud
* '''Datamodel''':
* '''ATT&CK''':
* '''Last Updated''': 2020-08-04
@@ -4274,7 +4274,7 @@ Detect and investigate dormant user accounts for your AWS environment that have
===Cloud cryptomining===
Monitor your cloud compute instances for activities related to cryptojacking/cryptomining. New instances that originate from previously unseen regions, users who launch abnormally high numbers of instances, or compute instances started by previously unseen users are just a few examples of potentially malicious behavior.
* '''Product''': Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud
* '''Product''': Splunk Security Analytics for AWS, Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud
* '''Datamodel''': Change
* '''ATT&CK''': [https://attack.mitre.org/techniques/T1078.004/ T1078.004], [https://attack.mitre.org/techniques/T1535/ T1535]
* '''Last Updated''': 2019-10-02
@@ -4331,7 +4331,7 @@ Monitor your cloud compute instances for activities related to cryptojacking/cry
===Cloud federated credential abuse===
This analytical story addresses events that indicate abuse of cloud federated credentials. These credentials are usually extracted from endpoint desktop or servers specially those servers that provide federation services such as Windows Active Directory Federation Services. Identity Federation relies on objects such as Oauth2 tokens, cookies or SAML assertions in order to provide seamless access between cloud and perimeter environments. If these objects are either hijacked or forged then attackers will be able to pivot into victim's cloud environements.
* '''Product''': Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud
* '''Product''': Splunk Security Analytics for AWS, Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud
* '''Datamodel''': Endpoint
* '''ATT&CK''': [https://attack.mitre.org/techniques/T1078/ T1078], [https://attack.mitre.org/techniques/T1003.001/ T1003.001], [https://attack.mitre.org/techniques/T1136.003/ T1136.003], [https://attack.mitre.org/techniques/T1556/ T1556], [https://attack.mitre.org/techniques/T1546.012/ T1546.012]
* '''Last Updated''': 2021-01-26
@@ -4673,7 +4673,7 @@ This story addresses detection and response around Sensitive Role usage within a
===Office 365 detections===
This story is focused around detecting Office 365 Attacks.
* '''Product''': Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud
* '''Product''': Splunk Security Analytics for AWS, Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud
* '''Datamodel''':
* '''ATT&CK''': [https://attack.mitre.org/techniques/T1110.001/ T1110.001], [https://attack.mitre.org/techniques/T1136.003/ T1136.003], [https://attack.mitre.org/techniques/T1562.007/ T1562.007], [https://attack.mitre.org/techniques/T1556/ T1556], [https://attack.mitre.org/techniques/T1110/ T1110], [https://attack.mitre.org/techniques/T1114/ T1114], [https://attack.mitre.org/techniques/T1114.003/ T1114.003], [https://attack.mitre.org/techniques/T1114.002/ T1114.002]
* '''Last Updated''': 2020-12-16
@@ -4831,7 +4831,7 @@ Use the searches in this Analytic Story to monitor your AWS EC2 instances for ev
===Suspicious aws login activities===
Monitor your AWS authentication events using your CloudTrail logs. Searches within this Analytic Story will help you stay aware of and investigate suspicious logins.
* '''Product''': Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud
* '''Product''': Splunk Security Analytics for AWS, Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud
* '''Datamodel''': Authentication
* '''ATT&CK''': [https://attack.mitre.org/techniques/T1535/ T1535], [https://attack.mitre.org/techniques/T1078.004/ T1078.004]
* '''Last Updated''': 2019-05-01
@@ -4886,7 +4886,7 @@ Monitor your AWS authentication events using your CloudTrail logs. Searches with
===Suspicious aws s3 activities===
Use the searches in this Analytic Story to monitor your AWS S3 buckets for evidence of anomalous activity and suspicious behaviors, such as detecting open S3 buckets and buckets being accessed from a new IP. The contextual and investigative searches will give you more information, when required.
* '''Product''': Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud
* '''Product''': Splunk Security Analytics for AWS, Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud
* '''Datamodel''':
* '''ATT&CK''': [https://attack.mitre.org/techniques/T1530/ T1530]
* '''Last Updated''': 2018-07-24
@@ -4975,7 +4975,7 @@ Leverage these searches to monitor your AWS network traffic for evidence of anom
===Suspicious cloud authentication activities===
Monitor your cloud authentication events. Searches within this Analytic Story leverage the recent cloud updates to the Authentication data model to help you stay aware of and investigate suspicious login activity.
* '''Product''': Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud
* '''Product''': Splunk Security Analytics for AWS, Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud
* '''Datamodel''': Authentication
* '''ATT&CK''': [https://attack.mitre.org/techniques/T1535/ T1535], [https://attack.mitre.org/techniques/T1078.004/ T1078.004]
* '''Last Updated''': 2020-06-04
@@ -5034,7 +5034,7 @@ Monitor your cloud authentication events. Searches within this Analytic Story le
===Suspicious cloud instance activities===
Monitor your cloud infrastructure provisioning activities for behaviors originating from unfamiliar or unusual locations. These behaviors may indicate that malicious activities are occurring somewhere within your cloud environment.
* '''Product''': Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud
* '''Product''': Splunk Security Analytics for AWS, Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud
* '''Datamodel''': Change
* '''ATT&CK''': [https://attack.mitre.org/techniques/T1078.004/ T1078.004]
* '''Last Updated''': 2020-08-25
@@ -5083,7 +5083,7 @@ Monitor your cloud infrastructure provisioning activities for behaviors originat
===Suspicious cloud provisioning activities===
Monitor your cloud infrastructure provisioning activities for behaviors originating from unfamiliar or unusual locations. These behaviors may indicate that malicious activities are occurring somewhere within your cloud environment.
* '''Product''': Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud
* '''Product''': Splunk Security Analytics for AWS, Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud
* '''Datamodel''': Change
* '''ATT&CK''': [https://attack.mitre.org/techniques/T1078/ T1078]
* '''Last Updated''': 2018-08-20
@@ -5132,7 +5132,7 @@ Monitor your cloud infrastructure provisioning activities for behaviors originat
===Suspicious cloud user activities===
Detect and investigate suspicious activities by users and roles in your cloud environments.
* '''Product''': Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud
* '''Product''': Splunk Security Analytics for AWS, Splunk Security Analytics for AWS, Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud
* '''Datamodel''': Change
* '''ATT&CK''': [https://attack.mitre.org/techniques/T1078.004/ T1078.004], [https://attack.mitre.org/techniques/T1078/ T1078]
* '''Last Updated''': 2020-09-04
@@ -6045,7 +6045,7 @@ Leverage searches that allow you to detect and investigate unusual activities th
===Ransomware cloud===
Leverage searches that allow you to detect and investigate unusual activities that might relate to ransomware. These searches include cloud related objects that may be targeted by malicious actors via cloud providers own encryption features.
* '''Product''': Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud
* '''Product''': Splunk Security Analytics for AWS, Splunk Enterprise, Splunk Enterprise Security, Splunk Cloud
* '''Datamodel''':
* '''ATT&CK''': [https://attack.mitre.org/techniques/T1486/ T1486]
* '''Last Updated''': 2020-10-27
@@ -6827,7 +6827,7 @@ Reduce the risk of CVE-2018-11409, an information disclosure vulnerability withi
''
#############
# Automatically generated by doc_gen.py in https://github.com/splunk/security_content
# On Date: 2021-03-29 18:42:22.600999 UTC
# On Date: 2021-04-02 17:10:21.639044 UTC
# Author: Splunk Security Research
# Contact: research@splunk.com
#############
package/README.md
+7
View File
@@ -0,0 +1,7 @@
# Splunk ES Content Update
This subscription service delivers pre-packaged Security Content for use with Splunk Enterprise Security. Subscribers get regular updates to help security practitioners more quickly address ongoing and time-sensitive customer problems and threats.
Requires Splunk Enterprise Security version 4.5 or greater.
For more information please visit the [Splunk ES Content Update user documentation](https://docs.splunk.com/Documentation/ESSOC).
package/README/essoc_story_detail.txt
+15
View File
@@ -0,0 +1,15 @@
The Analytic Story Details dashboard renders all the details of the content related to a specific analytic story which
can be chose via the drop down
Each analytic story has attributes associated with it and the following:
______________________________________________________________________
Analytic Story: name of the analytic story
Description ; description of the analytic story
Search Name : The name of the searches belonging to the chosen analytic story
Search : The search query which looks for an attack pattern corresponding to the analytic story
Search Description: The description of the search query
Asset Type: The analytic story specifies what asset in the infrastructure may be compromised
Category: The category that the search belongs to (malware, vulnerabilities, best practices, abuse)
Kill Chain Phase: The kill chain phase of the attack that the search is after.
package/README/essoc_summary.txt
+24
View File
@@ -0,0 +1,24 @@
The ES_SOC Summary Dashboard provides you a summarized view of the analytic story contents of the ES-SOC app.
The dashboard has the following panels gives you following details
1) Analytic story Summary
- Total Analytic Stories : The total number of Analytic stories in the ES-SOC application
- Total Searches: The total number of searches in ES-SOC
- Searches added last week: Number of searches added to ES-SOC in the last week.
2) Analytic story Category: This dashboard panel summarizes the categories of the searches that the ES-SOC app contains. The categories of the analytic stories are as follow
-Malware: These searches detect specific malware behavior for a particular phase of the attack kill chain. E.g. a malware’s delivery method via email or a malware’s installation behavior via registry key changes
-Vulnerability: These searches detect behavior or a signature of a vulnerable software in use. These searches are not designed to replace vulnerability management or scanning systems. The purpose of these searches is to discover a vulnerability through side effects or behaviors.
-Abuse: Some actions can be deemed malicious because they are unexpected, violate corporate policy or are significantly different than the actions of other users. E.g. A USB disk that is seen on multiple systems or a user that uploads excessive files to a cloud service or a database query that dumps an entire table
-Best Practices: Searches that correspond to specific guidelines from organizations like SANS or OWASP
3) Kill Chain phases: Every analytic story has one or more searches which look for a certain kind of attack pattern/behavior. These searches have an attribute which essentially tells you what Kill chain phase does the search correspond to.
The numbers on the dashboard represents the number of searches correponding to each kill chain phase
4) Analytic story table: This table gives the user a comprehensive view of some of the details of the analytic story. Some of the listed attributes are:
- Analytic Story : The name of the analytic story
- Description: The description of the analyttic story
- Search names: The name of the searches in each analytic story
- Datamodels: The name of the datamodel that the search is querying against.
- Technology Examples: This field represent some examples related to the technologies required to populate the datamodels(Nessues, Cisco Firewall,etc)
- Kill chain phase: The name of the kill chain phase that the search belongs to
package/README/essoc_usage_dashboard.txt
+51
View File
@@ -0,0 +1,51 @@
######################
ESSOC Usage Dashboard#
######################
The ESSOC Usage dashboard is designed to provide high-level insight into the usage of the ES-SOC app. It is suitable for display when providing feedback to the Splunk team or for identifying how the ES-SOC app is being used. This dashboard has two time selectors that work independently - the top time selector determines the search time range for all the single-value. And the lower time selector, determines the time range for the usage table.
IMPORTANT: The user loading this dashboard must have permission to search the _audit index
##################
#Dashboard panels#
##################
Searches Ran
The total number of searches in ES-SOC that were executed. This number includes scheduled searches and ad hoc searches run from the search bar using the '| savedsearch <ESSOC search_name> ‘ syntax
Unique Searches
The unique/distinct searches executed on the deployment. This is equivalent to the distinct count of searches run in the ES-SOC app.
Most Run
The total number of searches in ES-SOC that were executed. This number includes scheduled searches and ad hoc searches run from the search bar using the '| savedsearch <ESSOC search_name> ‘ syntax.
Ad hoc Searches
The total number of searches run from the search bar using the '| savedsearch <ESSOC search_name> ‘ syntax.
Scheduled
The total number of ESSOC searches run that were scheduled.
Most Active User
The user who executed the highest number/count of searches. This calculation includes scheduled searches and ad hoc searches run from the search bar using the '| savedsearch <ESSOC search_name> ‘ syntax.
Search Run Time (seconds)
Total run time of all searches executed in seconds. This calculation includes scheduled searches and ad hoc searches run from the search bar using the '| savedsearch <ESSOC search_name> ‘ syntax.
Average Run Time (seconds)
Average run time of all searches executed in seconds. This calculation includes scheduled searches and ad hoc searches run from the search bar using the '| savedsearch <ESSOC search_name> ‘ syntax.
Max Run Time (seconds)
The run time of the longest running search. This calculation includes scheduled searches and ad hoc searches run from the search bar using the '| savedsearch <ESSOC search_name> ‘ syntax.
Search summary
This table provides details on each search that was executed in the ESSOC app.
package/app.manifest
+46
View File
@@ -0,0 +1,46 @@
{
"schemaVersion": "1.0.0",
"info": {
"title": "ES Content Updates",
"id": {
"group": null,
"name": "DA-ESS-ContentUpdate",
"version": "3.19.0"
},
"author": [
{
"name": "Splunk Security Research Team",
"email": "research@splunk.com",
"company": "Splunk"
}
],
"releaseDate": null,
"description": "Explore the Analytic Stories included with ES Content Updates.",
"classification": {
"intendedAudience": null,
"categories": [],
"developmentStatus": null
},
"commonInformationModels": null,
"license": {
"name": null,
"text": null,
"uri": null
},
"privacyPolicy": {
"name": null,
"text": null,
"uri": null
},
"releaseNotes": {
"name": null,
"text": "./README.md",
"uri": null
}
},
"dependencies": null,
"tasks": null,
"inputGroups": null,
"incompatibleApps": null,
"platformRequirements": null
}
package/bin/dnstwist.py
Executable
+400
View File
@@ -0,0 +1,400 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
#
#
#
# Licensed under the Apache License, Version 2.0 (the "License"): you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
#
# Code modified from DNSTwist (https://github.com/elceef/dnstwist)
# Thanks elceef!
#
# Changes made:
# Just kept the DomainFuzz class and passing the domain to the fuzzer. Then added
# the Splunk specific code around it
#
from __future__ import absolute_import, division, print_function, unicode_literals
import sys
import re
import csv
import time
import os
from splunklib.searchcommands import dispatch, GeneratingCommand, \
Configuration, Option, Boolean
from splunk.clilib.bundle_paths import make_splunkhome_path
class DomainFuzz(object):
def __init__(self, domain):
self.domain, self.tld = self.__domain_tld(domain)
self.domains = []
self.qwerty = {
'1': '2q', '2': '3wq1', '3': '4ew2', '4': '5re3',
'5': '6tr4', '6': '7yt5', '7': '8uy6', '8': '9iu7',
'9': '0oi8', '0': 'po9', 'q': '12wa', 'w': '3esaq2',
'e': '4rdsw3', 'r': '5tfde4', 't': '6ygfr5', 'y': '7uhgt6',
'u': '8ijhy7', 'i': '9okju8', 'o': '0plki9', 'p': 'lo0',
'a': 'qwsz', 's': 'edxzaw', 'd': 'rfcxse', 'f': 'tgvcdr',
'g': 'yhbvft', 'h': 'ujnbgy', 'j': 'ikmnhu', 'k': 'olmji',
'l': 'kop', 'z': 'asx', 'x': 'zsdc', 'c': 'xdfv',
'v': 'cfgb', 'b': 'vghn', 'n': 'bhjm', 'm': 'njk'
}
self.qwertz = {
'1': '2q', '2': '3wq1', '3': '4ew2', '4': '5re3',
'5': '6tr4', '6': '7zt5', '7': '8uz6', '8': '9iu7',
'9': '0oi8', '0': 'po9', 'q': '12wa', 'w': '3esaq2',
'e': '4rdsw3', 'r': '5tfde4', 't': '6zgfr5',
'z': '7uhgt6', 'u': '8ijhz7', 'i': '9okju8',
'o': '0plki9', 'p': 'lo0', 'a': 'qwsy', 's': 'edxyaw',
'd': 'rfcxse', 'f': 'tgvcdr', 'g': 'zhbvft',
'h': 'ujnbgz', 'j': 'ikmnhu', 'k': 'olmji', 'l': 'kop',
'y': 'asx', 'x': 'ysdc', 'c': 'xdfv', 'v': 'cfgb',
'b': 'vghn', 'n': 'bhjm', 'm': 'njk'
}
self.azerty = {
'1': '2a', '2': '3za1', '3': '4ez2', '4': '5re3',
'5': '6tr4', '6': '7yt5', '7': '8uy6', '8': '9iu7',
'9': '0oi8', '0': 'po9', 'a': '2zq1', 'z': '3esqa2',
'e': '4rdsz3', 'r': '5tfde4', 't': '6ygfr5',
'y': '7uhgt6', 'u': '8ijhy7', 'i': '9okju8',
'o': '0plki9', 'p': 'lo0m', 'q': 'zswa', 's': 'edxwqz',
'd': 'rfcxse', 'f': 'tgvcdr', 'g': 'yhbvft',
'h': 'ujnbgy', 'j': 'iknhu', 'k': 'olji', 'l': 'kopm',
'm': 'lp', 'w': 'sxq', 'x': 'zsdc', 'c': 'xdfv',
'v': 'cfgb', 'b': 'vghn', 'n': 'bhj'
}
self.keyboards = [self.qwerty, self.qwertz, self.azerty]
def __domain_tld(self, domain):
domain = domain.rsplit('.', 2)
if len(domain) == 2:
return domain[0], domain[1]
return domain[0] + '.' + domain[1], domain[2]
def __validate_domain(self, domain):
if len(domain) == len(domain.encode('idna')) and domain != domain.encode('idna'):
return False
allowed = re.compile(b'(?=^.{4,253}$)(^((?!-)[a-zA-Z0-9-]{1,63}(?<!-)\\.)+[a-zA-Z]{2,63}\\.?$)', re.IGNORECASE)
return allowed.match(domain.encode('idna'))
def __filter_domains(self):
seen = set()
filtered = []
for d in self.domains:
# if not self.__validate_domain(d['domain-name']):
# p_err("debug: invalid domain %s\n" % d['domain-name'])
try:
if self.__validate_domain(d['domain-name']) and d['domain-name'] not in seen:
seen.add(d['domain-name'])
filtered.append(d)
except ValueError:
continue
self.domains = filtered
def __bitsquatting(self):
result = []
masks = [1, 2, 4, 8, 16, 32, 64, 128]
for i in range(0, len(self.domain)):
c = self.domain[i]
for j in range(0, len(masks)):
b = chr(ord(c) ^ masks[j])
o = ord(b)
if (o >= 48 and o <= 57) or (o >= 97 and o <= 122) or o == 45:
result.append(self.domain[:i] + b + self.domain[i+1:])
return result
def __homoglyph(self):
glyphs = {
'a': [u'à', u'á', u'â', u'ã', u'ä', u'å', u'ɑ', u'а', u'ạ', u'ǎ', u'ă', u'ȧ', u'ӓ'],
'b': ['d', 'lb', 'ib', u'ʙ', u'Ь', u'b̔', u'ɓ', u'Б'],
'c': [u'ϲ', u'с', u'ƈ', u'ċ', u'ć', u'ç'],
'd': ['b', 'cl', 'dl', 'di', u'ԁ', u'ժ', u'ɗ', u'đ'],
'e': [u'é', u'ê', u'ë', u'ē', u'ĕ', u'ě', u'ė', u'е', u'ẹ', u'ę', u'є', u'ϵ', u'ҽ'],
'f': [u'Ϝ', u'ƒ', u'Ғ'],
'g': ['q', u'ɢ', u'ɡ', u'Ԍ', u'Ԍ', u'ġ', u'ğ', u'ց', u'ǵ', u'ģ'],
'h': ['lh', 'ih', u'һ', u'հ', u'Ꮒ', u'н'],
'i': ['1', 'l', u'Ꭵ', u'í', u'ï', u'ı', u'ɩ', u'ι', u'ꙇ', u'ǐ', u'ĭ'],
'j': [u'ј', u'ʝ', u'ϳ', u'ɉ'],
'k': ['lk', 'ik', 'lc', u'κ', u'ⲕ', u'κ'],
'l': ['1', 'i', u'ɫ', u'ł'],
'm': ['n', 'nn', 'rn', 'rr', u'ṃ', u'ᴍ', u'м', u'ɱ'],
'n': ['m', 'r', u'ń'],
'o': ['0', u'Ο', u'ο', u'О', u'о', u'Օ', u'ȯ', u'ọ', u'ỏ', u'ơ', u'ó', u'ö', u'ӧ'],
'p': [u'ρ', u'р', u'ƿ', u'Ϸ', u'Þ'],
'q': ['g', u'զ', u'ԛ', u'գ', u'ʠ'],
'r': [u'ʀ', u'Г', u'ᴦ', u'ɼ', u'ɽ'],
's': [u'Ⴝ', u'Ꮪ', u'ʂ', u'ś', u'ѕ'],
't': [u'τ', u'т', u'ţ'],
'u': [u'μ', u'υ', u'Ս', u'ս', u'ц', u'ᴜ', u'ǔ', u'ŭ'],
'v': [u'ѵ', u'ν', u'v̇'],
'w': ['vv', u'ѡ', u'ա', u'ԝ'],
'x': [u'х', u'ҳ', u'ẋ'],
'y': [u'ʏ', u'γ', u'у', u'Ү', u'ý'],
'z': [u'ʐ', u'ż', u'ź', u'ʐ', u'ᴢ']
}
result = []
for ws in range(0, len(self.domain)):
for i in range(0, (len(self.domain)-ws)+1):
win = self.domain[i:i+ws]
j = 0
while j < ws:
c = win[j]
if c in glyphs:
win_copy = win
for g in glyphs[c]:
win = win.replace(c, g)
result.append(self.domain[:i] + win + self.domain[i+ws:])
win = win_copy
j += 1
return list(set(result))
def __hyphenation(self):
result = []
for i in range(1, len(self.domain)):
result.append(self.domain[:i] + '-' + self.domain[i:])
return result
def __insertion(self):
result = []
for i in range(1, len(self.domain)-1):
for keys in self.keyboards:
if self.domain[i] in keys:
for c in keys[self.domain[i]]:
result.append(self.domain[:i] + c + self.domain[i] + self.domain[i+1:])
result.append(self.domain[:i] + self.domain[i] + c + self.domain[i+1:])
return list(set(result))
def __omission(self):
result = []
for i in range(0, len(self.domain)):
result.append(self.domain[:i] + self.domain[i+1:])
n = re.sub(r'(.)\1+', r'\1', self.domain)
if n not in result and n != self.domain:
result.append(n)
return list(set(result))
def __repetition(self):
result = []
for i in range(0, len(self.domain)):
if self.domain[i].isalpha():
result.append(self.domain[:i] + self.domain[i] + self.domain[i] + self.domain[i+1:])
return list(set(result))
def __replacement(self):
result = []
for i in range(0, len(self.domain)):
for keys in self.keyboards:
if self.domain[i] in keys:
for c in keys[self.domain[i]]:
result.append(self.domain[:i] + c + self.domain[i+1:])
return list(set(result))
def __subdomain(self):
result = []
for i in range(1, len(self.domain)):
if self.domain[i] not in ['-', '.'] and self.domain[i-1] not in ['-', '.']:
result.append(self.domain[:i] + '.' + self.domain[i:])
return result
def __transposition(self):
result = []
for i in range(0, len(self.domain)-1):
if self.domain[i+1] != self.domain[i]:
result.append(self.domain[:i] + self.domain[i+1] + self.domain[i] + self.domain[i+2:])
return result
def __vowel_swap(self):
vowels = 'aeiou'
result = []
for i in range(0, len(self.domain)):
for vowel in vowels:
if self.domain[i] in vowels:
result.append(self.domain[:i] + vowel + self.domain[i+1:])
return list(set(result))
def __addition(self):
result = []
for i in range(97, 123):
result.append(self.domain + chr(i))
return result
def generate(self):
self.domains.append({'fuzzer': 'Original*', 'domain-name': self.domain + '.' + self.tld})
for domain in self.__addition():
self.domains.append({'fuzzer': 'Addition', 'domain-name': domain + '.' + self.tld})
for domain in self.__bitsquatting():
self.domains.append({'fuzzer': 'Bitsquatting', 'domain-name': domain + '.' + self.tld})
for domain in self.__homoglyph():
self.domains.append({'fuzzer': 'Homoglyph', 'domain-name': domain + '.' + self.tld})
for domain in self.__hyphenation():
self.domains.append({'fuzzer': 'Hyphenation', 'domain-name': domain + '.' + self.tld})
for domain in self.__insertion():
self.domains.append({'fuzzer': 'Insertion', 'domain-name': domain + '.' + self.tld})
for domain in self.__omission():
self.domains.append({'fuzzer': 'Omission', 'domain-name': domain + '.' + self.tld})
for domain in self.__repetition():
self.domains.append({'fuzzer': 'Repetition', 'domain-name': domain + '.' + self.tld})
for domain in self.__replacement():
self.domains.append({'fuzzer': 'Replacement', 'domain-name': domain + '.' + self.tld})
for domain in self.__subdomain():
self.domains.append({'fuzzer': 'Subdomain', 'domain-name': domain + '.' + self.tld})
for domain in self.__transposition():
self.domains.append({'fuzzer': 'Transposition', 'domain-name': domain + '.' + self.tld})
for domain in self.__vowel_swap():
self.domains.append({'fuzzer': 'Vowel-swap', 'domain-name': domain + '.' + self.tld})
if not self.domain.startswith('www.'):
self.domains.append({'fuzzer': 'Various', 'domain-name': 'ww' + self.domain + '.' + self.tld})
self.domains.append({'fuzzer': 'Various', 'domain-name': 'www' + self.domain + '.' + self.tld})
self.domains.append({'fuzzer': 'Various', 'domain-name': 'www-' + self.domain + '.' + self.tld})
if '.' in self.tld:
self.domains.append({'fuzzer': 'Various', 'domain-name': self.domain + '.' + self.tld.split('.')[-1]})
self.domains.append({'fuzzer': 'Various', 'domain-name': self.domain + self.tld})
if '.' not in self.tld:
self.domains.append({'fuzzer': 'Various', 'domain-name': self.domain + self.tld + '.' + self.tld})
if self.tld != 'com' and '.' not in self.tld:
self.domains.append({'fuzzer': 'Various', 'domain-name': self.domain + '-' + self.tld + '.com'})
self.__filter_domains()
@Configuration(distributed=True)
class DnsTwistCommand(GeneratingCommand):
domainlist_file_name = Option(doc='''
**Syntax:** **domainlist=***<path>*
**Description:** CSV file from which repeated random samples will be drawn
''', name='domainlist', require=False)
populate_from_cim = Option(doc='''
**Syntax: populate_cim=<bool>
**Description:** When `true`, populates Splunk_SA_CIM lookups cim_corporate_email_domains.csv
and cim_corporate_web_domains.csv with dnstwisted domains. Defaults to `false`.
''', name='populate_from_cim', default=False, validate=Boolean())
domain = Option(doc='''
**Syntax:** **domain=***<domain name>*
**Description:** Domain to DNS generated twisted entries for.
''', name='domain', require=False, default='')
def generate(self):
event_count = 0
csv_file_names = []
if self.populate_from_cim:
csv_file_names.append(make_splunkhome_path([
'etc',
'apps',
'Splunk_SA_CIM',
'lookups',
'cim_corporate_email_domains.csv']))
csv_file_names.append(make_splunkhome_path([
'etc',
'apps',
'Splunk_SA_CIM',
'lookups',
'cim_corporate_web_domains.csv']))
# Make sure we just get the base file name from file. In case there was some directory traversal going on.
if self.domainlist_file_name:
sanitized_file_name = os.path.basename(self.domainlist_file_name)
lookup_path = make_splunkhome_path(['etc', 'apps', 'DA-ESS-ContentUpdate', 'lookups', sanitized_file_name])
# Make sure there really isn't any directory traversal going on.
valid_path = True
if "../" in lookup_path:
valid_path = False
# Make sure the path that is created by adding the file name to the path is the same as the
# absolute path
if lookup_path != os.path.abspath(lookup_path):
valid_path = False
if valid_path:
csv_file_names.append(lookup_path)
domains_to_twist = []
for csv_file_name in csv_file_names:
if os.path.exists(csv_file_name):
# this is nasty but works .. please forgive me
if sys.version_info >= (3, 0):
csv_file = open(csv_file_name, "r", newline='')
else:
csv_file = open(csv_file_name, "r")
for input_domain in csv.DictReader(csv_file):
if input_domain['domain'] not in domains_to_twist:
domains_to_twist.append(input_domain['domain'])
# if a single domain is passed lets just calculate that
if self.domain != '':
domains_to_twist = []
domains_to_twist.append(self.domain)
for domain_to_twist in domains_to_twist:
domain_to_twist = domain_to_twist.lstrip('*')
dfuzz = DomainFuzz(domain_to_twist)
dfuzz.generate()
domains = dfuzz.domains
for domain in domains:
# We don't want to keep the original domain
if domain['domain-name'] in domain_to_twist:
continue
event_count += 1
yield {
'_time': time.time(),
'event_no': event_count,
'_raw': domain['domain-name'],
'domain': '*'+domain['domain-name']+'*',
'original_domain': domain_to_twist
}
def __init__(self):
super(DnsTwistCommand, self).__init__()
dispatch(DnsTwistCommand, sys.argv, sys.stdin, sys.stdout, __name__)
package/bin/escu_contextualize.py
+1
View File
@@ -0,0 +1 @@
# dropped AR action support due to python 3 dependency, we leverage playbooks in stories as an alternative.
package/bin/escu_investigate.py
+1
View File
@@ -0,0 +1 @@
# dropped AR action support due to python 3 dependency, we leverage playbooks in stories as an alternative.
package/bin/runstory.py
+1
View File
@@ -0,0 +1 @@
# runstory was deprecated, its functionality was moved to: https://github.com/splunk/analytic_story_execution
package/bin/splunklib/__init__.py
Executable
+19
View File
@@ -0,0 +1,19 @@
# Copyright 2011-2015 Splunk, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"): you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
"""Python library for Splunk."""
__version_info__ = (1, 6, 2)
__version__ = ".".join(map(str, __version_info__))
package/bin/splunklib/binding.py
Executable
+1373
View File
File diff suppressed because it is too large Load Diff
package/bin/splunklib/client.py
Executable
+3718
View File
File diff suppressed because it is too large Load Diff
package/bin/splunklib/data.py
Executable
+258
View File
@@ -0,0 +1,258 @@
# Copyright 2011-2015 Splunk, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"): you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
"""The **splunklib.data** module reads the responses from splunkd in Atom Feed
format, which is the format used by most of the REST API.
"""
from xml.etree.ElementTree import XML
__all__ = ["load"]
# LNAME refers to element names without namespaces; XNAME is the same
# name, but with an XML namespace.
LNAME_DICT = "dict"
LNAME_ITEM = "item"
LNAME_KEY = "key"
LNAME_LIST = "list"
XNAMEF_REST = "{http://dev.splunk.com/ns/rest}%s"
XNAME_DICT = XNAMEF_REST % LNAME_DICT
XNAME_ITEM = XNAMEF_REST % LNAME_ITEM
XNAME_KEY = XNAMEF_REST % LNAME_KEY
XNAME_LIST = XNAMEF_REST % LNAME_LIST
# Some responses don't use namespaces (eg: search/parse) so we look for
# both the extended and local versions of the following names.
def isdict(name):
return name == XNAME_DICT or name == LNAME_DICT
def isitem(name):
return name == XNAME_ITEM or name == LNAME_ITEM
def iskey(name):
return name == XNAME_KEY or name == LNAME_KEY
def islist(name):
return name == XNAME_LIST or name == LNAME_LIST
def hasattrs(element):
return len(element.attrib) > 0
def localname(xname):
rcurly = xname.find('}')
return xname if rcurly == -1 else xname[rcurly+1:]
def load(text, match=None):
"""This function reads a string that contains the XML of an Atom Feed, then
returns the
data in a native Python structure (a ``dict`` or ``list``). If you also
provide a tag name or path to match, only the matching sub-elements are
loaded.
:param text: The XML text to load.
:type text: ``string``
:param match: A tag name or path to match (optional).
:type match: ``string``
"""
if text is None: return None
text = text.strip()
if len(text) == 0: return None
nametable = {
'namespaces': [],
'names': {}
}
root = XML(text)
items = [root] if match is None else root.findall(match)
count = len(items)
if count == 0:
return None
elif count == 1:
return load_root(items[0], nametable)
else:
return [load_root(item, nametable) for item in items]
# Load the attributes of the given element.
def load_attrs(element):
if not hasattrs(element): return None
attrs = record()
for key, value in element.attrib.iteritems():
attrs[key] = value
return attrs
# Parse a <dict> element and return a Python dict
def load_dict(element, nametable = None):
value = record()
children = list(element)
for child in children:
assert iskey(child.tag)
name = child.attrib["name"]
value[name] = load_value(child, nametable)
return value
# Loads the given elements attrs & value into single merged dict.
def load_elem(element, nametable=None):
name = localname(element.tag)
attrs = load_attrs(element)
value = load_value(element, nametable)
if attrs is None: return name, value
if value is None: return name, attrs
# If value is simple, merge into attrs dict using special key
if isinstance(value, str):
attrs["$text"] = value
return name, attrs
# Both attrs & value are complex, so merge the two dicts, resolving collisions.
collision_keys = []
for key, val in attrs.iteritems():
if key in value and key in collision_keys:
value[key].append(val)
elif key in value and key not in collision_keys:
value[key] = [value[key], val]
collision_keys.append(key)
else:
value[key] = val
return name, value
# Parse a <list> element and return a Python list
def load_list(element, nametable=None):
assert islist(element.tag)
value = []
children = list(element)
for child in children:
assert isitem(child.tag)
value.append(load_value(child, nametable))
return value
# Load the given root element.
def load_root(element, nametable=None):
tag = element.tag
if isdict(tag): return load_dict(element, nametable)
if islist(tag): return load_list(element, nametable)
k, v = load_elem(element, nametable)
return Record.fromkv(k, v)
# Load the children of the given element.
def load_value(element, nametable=None):
children = list(element)
count = len(children)
# No children, assume a simple text value
if count == 0:
text = element.text
if text is None:
return None
text = text.strip()
if len(text) == 0:
return None
return text
# Look for the special case of a single well-known structure
if count == 1:
child = children[0]
tag = child.tag
if isdict(tag): return load_dict(child, nametable)
if islist(tag): return load_list(child, nametable)
value = record()
for child in children:
name, item = load_elem(child, nametable)
# If we have seen this name before, promote the value to a list
if value.has_key(name):
current = value[name]
if not isinstance(current, list):
value[name] = [current]
value[name].append(item)
else:
value[name] = item
return value
# A generic utility that enables "dot" access to dicts
class Record(dict):
"""This generic utility class enables dot access to members of a Python
dictionary.
Any key that is also a valid Python identifier can be retrieved as a field.
So, for an instance of ``Record`` called ``r``, ``r.key`` is equivalent to
``r['key']``. A key such as ``invalid-key`` or ``invalid.key`` cannot be
retrieved as a field, because ``-`` and ``.`` are not allowed in
identifiers.
Keys of the form ``a.b.c`` are very natural to write in Python as fields. If
a group of keys shares a prefix ending in ``.``, you can retrieve keys as a
nested dictionary by calling only the prefix. For example, if ``r`` contains
keys ``'foo'``, ``'bar.baz'``, and ``'bar.qux'``, ``r.bar`` returns a record
with the keys ``baz`` and ``qux``. If a key contains multiple ``.``, each
one is placed into a nested dictionary, so you can write ``r.bar.qux`` or
``r['bar.qux']`` interchangeably.
"""
sep = '.'
def __call__(self, *args):
if len(args) == 0: return self
return Record((key, self[key]) for key in args)
def __getattr__(self, name):
try:
return self[name]
except KeyError:
raise AttributeError(name)
def __delattr__(self, name):
del self[name]
def __setattr__(self, name, value):
self[name] = value
@staticmethod
def fromkv(k, v):
result = record()
result[k] = v
return result
def __getitem__(self, key):
if key in self:
return dict.__getitem__(self, key)
key += self.sep
result = record()
for k,v in self.iteritems():
if not k.startswith(key):
continue
suffix = k[len(key):]
if '.' in suffix:
ks = suffix.split(self.sep)
z = result
for x in ks[:-1]:
if x not in z:
z[x] = record()
z = z[x]
z[ks[-1]] = v
else:
result[suffix] = v
if len(result) == 0:
raise KeyError("No key or prefix: %s" % key)
return result
def record(value=None):
"""This function returns a :class:`Record` instance constructed with an
initial value that you provide.
:param `value`: An initial record value.
:type `value`: ``dict``
"""
if value is None: value = {}
return Record(value)
package/bin/splunklib/modularinput/__init__.py
Executable
+12
View File
@@ -0,0 +1,12 @@
"""The following imports allow these classes to be imported via
the splunklib.modularinput package like so:
from splunklib.modularinput import *
"""
from .argument import Argument
from .event import Event
from .event_writer import EventWriter
from .input_definition import InputDefinition
from .scheme import Scheme
from .script import Script
from .validation_definition import ValidationDefinition
package/bin/splunklib/modularinput/argument.py
Executable
+102
View File
@@ -0,0 +1,102 @@
# Copyright 2011-2015 Splunk, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"): you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
try:
import xml.etree.ElementTree as ET
except ImportError:
import xml.etree.cElementTree as ET
class Argument(object):
"""Class representing an argument to a modular input kind.
``Argument`` is meant to be used with ``Scheme`` to generate an XML
definition of the modular input kind that Splunk understands.
``name`` is the only required parameter for the constructor.
**Example with least parameters**::
arg1 = Argument(name="arg1")
**Example with all parameters**::
arg2 = Argument(
name="arg2",
description="This is an argument with lots of parameters",
validation="is_pos_int('some_name')",
data_type=Argument.data_type_number,
required_on_edit=True,
required_on_create=True
)
"""
# Constant values, do not change.
# These should be used for setting the value of an Argument object's data_type field.
data_type_boolean = "BOOLEAN"
data_type_number = "NUMBER"
data_type_string = "STRING"
def __init__(self, name, description=None, validation=None,
data_type=data_type_string, required_on_edit=False, required_on_create=False, title=None):
"""
:param name: ``string``, identifier for this argument in Splunk.
:param description: ``string``, human-readable description of the argument.
:param validation: ``string`` specifying how the argument should be validated, if using internal validation.
If using external validation, this will be ignored.
:param data_type: ``string``, data type of this field; use the class constants.
"data_type_boolean", "data_type_number", or "data_type_string".
:param required_on_edit: ``Boolean``, whether this arg is required when editing an existing modular input of this kind.
:param required_on_create: ``Boolean``, whether this arg is required when creating a modular input of this kind.
:param title: ``String``, a human-readable title for the argument.
"""
self.name = name
self.description = description
self.validation = validation
self.data_type = data_type
self.required_on_edit = required_on_edit
self.required_on_create = required_on_create
self.title = title
def add_to_document(self, parent):
"""Adds an ``Argument`` object to this ElementTree document.
Adds an <arg> subelement to the parent element, typically <args>
and sets up its subelements with their respective text.
:param parent: An ``ET.Element`` to be the parent of a new <arg> subelement
:returns: An ``ET.Element`` object representing this argument.
"""
arg = ET.SubElement(parent, "arg")
arg.set("name", self.name)
if self.title is not None:
ET.SubElement(arg, "title").text = self.title
if self.description is not None:
ET.SubElement(arg, "description").text = self.description
if self.validation is not None:
ET.SubElement(arg, "validation").text = self.validation
# add all other subelements to this Argument, represented by (tag, text)
subelements = [
("data_type", self.data_type),
("required_on_edit", self.required_on_edit),
("required_on_create", self.required_on_create)
]
for name, value in subelements:
ET.SubElement(arg, name).text = str(value).lower()
return arg
package/bin/splunklib/modularinput/event.py
Executable
+107
View File
@@ -0,0 +1,107 @@
# Copyright 2011-2015 Splunk, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"): you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
try:
import xml.etree.cElementTree as ET
except ImportError as ie:
import xml.etree.ElementTree as ET
class Event(object):
"""Represents an event or fragment of an event to be written by this modular input to Splunk.
To write an input to a stream, call the ``write_to`` function, passing in a stream.
"""
def __init__(self, data=None, stanza=None, time=None, host=None, index=None, source=None,
sourcetype=None, done=True, unbroken=True):
"""There are no required parameters for constructing an Event
**Example with minimal configuration**::
my_event = Event(
data="This is a test of my new event.",
stanza="myStanzaName",
time="%.3f" % 1372187084.000
)
**Example with full configuration**::
excellent_event = Event(
data="This is a test of my excellent event.",
stanza="excellenceOnly",
time="%.3f" % 1372274622.493,
host="localhost",
index="main",
source="Splunk",
sourcetype="misc",
done=True,
unbroken=True
)
:param data: ``string``, the event's text.
:param stanza: ``string``, name of the input this event should be sent to.
:param time: ``float``, time in seconds, including up to 3 decimal places to represent milliseconds.
:param host: ``string``, the event's host, ex: localhost.
:param index: ``string``, the index this event is specified to write to, or None if default index.
:param source: ``string``, the source of this event, or None to have Splunk guess.
:param sourcetype: ``string``, source type currently set on this event, or None to have Splunk guess.
:param done: ``boolean``, is this a complete ``Event``? False if an ``Event`` fragment.
:param unbroken: ``boolean``, Is this event completely encapsulated in this ``Event`` object?
"""
self.data = data
self.done = done
self.host = host
self.index = index
self.source = source
self.sourceType = sourcetype
self.stanza = stanza
self.time = time
self.unbroken = unbroken
def write_to(self, stream):
"""Write an XML representation of self, an ``Event`` object, to the given stream.
The ``Event`` object will only be written if its data field is defined,
otherwise a ``ValueError`` is raised.
:param stream: stream to write XML to.
"""
if self.data is None:
raise ValueError("Events must have at least the data field set to be written to XML.")
event = ET.Element("event")
if self.stanza is not None:
event.set("stanza", self.stanza)
event.set("unbroken", str(int(self.unbroken)))
# if a time isn't set, let Splunk guess by not creating a <time> element
if self.time is not None:
ET.SubElement(event, "time").text = str(self.time)
# add all other subelements to this Event, represented by (tag, text)
subelements = [
("source", self.source),
("sourcetype", self.sourceType),
("index", self.index),
("host", self.host),
("data", self.data)
]
for node, value in subelements:
if value is not None:
ET.SubElement(event, node).text = value
if self.done:
ET.SubElement(event, "done")
stream.write(ET.tostring(event))
stream.flush()
package/bin/splunklib/modularinput/event_writer.py
Executable
+84
View File
@@ -0,0 +1,84 @@
# Copyright 2011-2015 Splunk, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"): you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import sys
from .event import ET
try:
from cStringIO import StringIO
except ImportError:
from StringIO import StringIO
class EventWriter(object):
"""``EventWriter`` writes events and error messages to Splunk from a modular input.
Its two important methods are ``writeEvent``, which takes an ``Event`` object,
and ``log``, which takes a severity and an error message.
"""
# Severities that Splunk understands for log messages from modular inputs.
# Do not change these
DEBUG = "DEBUG"
INFO = "INFO"
WARN = "WARN"
ERROR = "ERROR"
FATAL = "FATAL"
def __init__(self, output = sys.stdout, error = sys.stderr):
"""
:param output: Where to write the output; defaults to sys.stdout.
:param error: Where to write any errors; defaults to sys.stderr.
"""
self._out = output
self._err = error
# has the opening <stream> tag been written yet?
self.header_written = False
def write_event(self, event):
"""Writes an ``Event`` object to Splunk.
:param event: An ``Event`` object.
"""
if not self.header_written:
self._out.write("<stream>")
self.header_written = True
event.write_to(self._out)
def log(self, severity, message):
"""Logs messages about the state of this modular input to Splunk.
These messages will show up in Splunk's internal logs.
:param severity: ``string``, severity of message, see severities defined as class constants.
:param message: ``string``, message to log.
"""
self._err.write("%s %s\n" % (severity, message))
self._err.flush()
def write_xml_document(self, document):
"""Writes a string representation of an
``ElementTree`` object to the output stream.
:param document: An ``ElementTree`` object.
"""
self._out.write(ET.tostring(document))
self._out.flush()
def close(self):
"""Write the closing </stream> tag to make this XML well formed."""
self._out.write("</stream>")
package/bin/splunklib/modularinput/input_definition.py
Executable
+59
View File
@@ -0,0 +1,59 @@
# Copyright 2011-2015 Splunk, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"): you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
try:
import xml.etree.cElementTree as ET
except ImportError as ie:
import xml.etree.ElementTree as ET
from .utils import parse_xml_data
class InputDefinition:
"""``InputDefinition`` encodes the XML defining inputs that Splunk passes to
a modular input script.
**Example**::
i = InputDefinition()
"""
def __init__ (self):
self.metadata = {}
self.inputs = {}
def __eq__(self, other):
if not isinstance(other, InputDefinition):
return False
return self.metadata == other.metadata and self.inputs == other.inputs
@staticmethod
def parse(stream):
"""Parse a stream containing XML into an ``InputDefinition``.
:param stream: stream containing XML to parse.
:return: definition: an ``InputDefinition`` object.
"""
definition = InputDefinition()
# parse XML from the stream, then get the root node
root = ET.parse(stream).getroot()
for node in root:
if node.tag == "configuration":
# get config for each stanza
definition.inputs = parse_xml_data(node, "stanza")
else:
definition.metadata[node.tag] = node.text
return definition
package/bin/splunklib/modularinput/scheme.py
Executable
+84
View File
@@ -0,0 +1,84 @@
# Copyright 2011-2015 Splunk, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"): you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
try:
import xml.etree.cElementTree as ET
except ImportError:
import xml.etree.ElementTree as ET
class Scheme(object):
"""Class representing the metadata for a modular input kind.
A ``Scheme`` specifies a title, description, several options of how Splunk should run modular inputs of this
kind, and a set of arguments which define a particular modular input's properties.
The primary use of ``Scheme`` is to abstract away the construction of XML to feed to Splunk.
"""
# Constant values, do not change
# These should be used for setting the value of a Scheme object's streaming_mode field.
streaming_mode_simple = "SIMPLE"
streaming_mode_xml = "XML"
def __init__(self, title):
"""
:param title: ``string`` identifier for this Scheme in Splunk.
"""
self.title = title
self.description = None
self.use_external_validation = True
self.use_single_instance = False
self.streaming_mode = Scheme.streaming_mode_xml
# list of Argument objects, each to be represented by an <arg> tag
self.arguments = []
def add_argument(self, arg):
"""Add the provided argument, ``arg``, to the ``self.arguments`` list.
:param arg: An ``Argument`` object to add to ``self.arguments``.
"""
self.arguments.append(arg)
def to_xml(self):
"""Creates an ``ET.Element`` representing self, then returns it.
:returns root, an ``ET.Element`` representing this scheme.
"""
root = ET.Element("scheme")
ET.SubElement(root, "title").text = self.title
# add a description subelement if it's defined
if self.description is not None:
ET.SubElement(root, "description").text = self.description
# add all other subelements to this Scheme, represented by (tag, text)
subelements = [
("use_external_validation", self.use_external_validation),
("use_single_instance", self.use_single_instance),
("streaming_mode", self.streaming_mode)
]
for name, value in subelements:
ET.SubElement(root, name).text = str(value).lower()
endpoint = ET.SubElement(root, "endpoint")
args = ET.SubElement(endpoint, "args")
# add arguments as subelements to the <args> element
for arg in self.arguments:
arg.add_to_document(args)
return root
package/bin/splunklib/modularinput/script.py
Executable
+176
View File
@@ -0,0 +1,176 @@
# Copyright 2011-2015 Splunk, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"): you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from abc import ABCMeta, abstractmethod
from urlparse import urlsplit
import sys
from ..client import Service
from .event_writer import EventWriter
from .input_definition import InputDefinition
from .validation_definition import ValidationDefinition
try:
import xml.etree.cElementTree as ET
except ImportError:
import xml.etree.ElementTree as ET
class Script(object):
"""An abstract base class for implementing modular inputs.
Subclasses should override ``get_scheme``, ``stream_events``,
and optionally ``validate_input`` if the modular input uses
external validation.
The ``run`` function is used to run modular inputs; it typically should
not be overridden.
"""
__metaclass__ = ABCMeta
def __init__(self):
self._input_definition = None
self._service = None
def run(self, args):
"""Runs this modular input
:param args: List of command line arguments passed to this script.
:returns: An integer to be used as the exit value of this program.
"""
# call the run_script function, which handles the specifics of running
# a modular input
return self.run_script(args, EventWriter(), sys.stdin)
def run_script(self, args, event_writer, input_stream):
"""Handles all the specifics of running a modular input
:param args: List of command line arguments passed to this script.
:param event_writer: An ``EventWriter`` object for writing events.
:param input_stream: An input stream for reading inputs.
:returns: An integer to be used as the exit value of this program.
"""
try:
if len(args) == 1:
# This script is running as an input. Input definitions will be
# passed on stdin as XML, and the script will write events on
# stdout and log entries on stderr.
self._input_definition = InputDefinition.parse(input_stream)
self.stream_events(self._input_definition, event_writer)
event_writer.close()
return 0
elif str(args[1]).lower() == "--scheme":
# Splunk has requested XML specifying the scheme for this
# modular input Return it and exit.
scheme = self.get_scheme()
if scheme is None:
event_writer.log(
EventWriter.FATAL,
"Modular input script returned a null scheme.")
return 1
else:
event_writer.write_xml_document(scheme.to_xml())
return 0
elif args[1].lower() == "--validate-arguments":
validation_definition = ValidationDefinition.parse(input_stream)
try:
self.validate_input(validation_definition)
return 0
except Exception as e:
root = ET.Element("error")
ET.SubElement(root, "message").text = str(e)
event_writer.write_xml_document(root)
return 1
else:
err_string = "ERROR Invalid arguments to modular input script:" + ' '.join(
args)
event_writer._err.write(err_string)
except Exception as e:
err_string = EventWriter.ERROR + str(e.message)
event_writer._err.write(err_string)
return 1
@property
def service(self):
""" Returns a Splunk service object for this script invocation.
The service object is created from the Splunkd URI and session key
passed to the command invocation on the modular input stream. It is
available as soon as the :code:`Script.stream_events` method is
called.
:return: :class:splunklib.client.Service. A value of None is returned,
if you call this method before the :code:`Script.stream_events` method
is called.
"""
if self._service is not None:
return self._service
if self._input_definition is None:
return None
splunkd_uri = self._input_definition.metadata["server_uri"]
session_key = self._input_definition.metadata["session_key"]
splunkd = urlsplit(splunkd_uri, allow_fragments=False)
self._service = Service(
scheme=splunkd.scheme,
host=splunkd.hostname,
port=splunkd.port,
token=session_key,
)
return self._service
@abstractmethod
def get_scheme(self):
"""The scheme defines the parameters understood by this modular input.
:return: a ``Scheme`` object representing the parameters for this modular input.
"""
def validate_input(self, definition):
"""Handles external validation for modular input kinds.
When Splunk calls a modular input script in validation mode, it will
pass in an XML document giving information about the Splunk instance (so
you can call back into it if needed) and the name and parameters of the
proposed input.
If this function does not throw an exception, the validation is assumed
to succeed. Otherwise any errors thrown will be turned into a string and
logged back to Splunk.
The default implementation always passes.
:param definition: The parameters for the proposed input passed by splunkd.
"""
pass
@abstractmethod
def stream_events(self, inputs, ew):
"""The method called to stream events into Splunk. It should do all of its output via
EventWriter rather than assuming that there is a console attached.
:param inputs: An ``InputDefinition`` object.
:param ew: An object with methods to write events and log messages to Splunk.
"""
package/bin/splunklib/modularinput/utils.py
Executable
+72
View File
@@ -0,0 +1,72 @@
# Copyright 2011-2015 Splunk, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"): you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
# File for utility functions
def xml_compare(expected, found):
"""Checks equality of two ``ElementTree`` objects.
:param expected: An ``ElementTree`` object.
:param found: An ``ElementTree`` object.
:return: ``Boolean``, whether the two objects are equal.
"""
# if comparing the same ET object
if expected == found:
return True
# compare element attributes, ignoring order
if set(expected.items()) != set(found.items()):
return False
# check for equal number of children
expected_children = list(expected)
found_children = list(found)
if len(expected_children) != len(found_children):
return False
# compare children
if not all([xml_compare(a, b) for a, b in zip(expected_children, found_children)]):
return False
# compare elements, if there is no text node, return True
if (expected.text is None or expected.text.strip() == "") \
and (found.text is None or found.text.strip() == ""):
return True
else:
return expected.tag == found.tag and expected.text == found.text \
and expected.attrib == found.attrib
def parse_parameters(param_node):
if param_node.tag == "param":
return param_node.text
elif param_node.tag == "param_list":
parameters = []
for mvp in param_node:
parameters.append(mvp.text)
return parameters
else:
raise ValueError("Invalid configuration scheme, %s tag unexpected." % param_node.tag)
def parse_xml_data(parent_node, child_node_tag):
data = {}
for child in parent_node:
if child.tag == child_node_tag:
if child_node_tag == "stanza":
data[child.get("name")] = {}
for param in child:
data[child.get("name")][param.get("name")] = parse_parameters(param)
elif "item" == parent_node.tag:
data[child.get("name")] = parse_parameters(child)
return data
package/bin/splunklib/modularinput/validation_definition.py
Executable
+83
View File
@@ -0,0 +1,83 @@
# Copyright 2011-2015 Splunk, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"): you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
try:
import xml.etree.cElementTree as ET
except ImportError as ie:
import xml.etree.ElementTree as ET
from .utils import parse_xml_data
class ValidationDefinition(object):
"""This class represents the XML sent by Splunk for external validation of a
new modular input.
**Example**::
``v = ValidationDefinition()``
"""
def __init__(self):
self.metadata = {}
self.parameters = {}
def __eq__(self, other):
if not isinstance(other, ValidationDefinition):
return False
return self.metadata == other.metadata and self.parameters == other.parameters
@staticmethod
def parse(stream):
"""Creates a ``ValidationDefinition`` from a provided stream containing XML.
The XML typically will look like this:
``<items>``
`` <server_host>myHost</server_host>``
`` <server_uri>https://127.0.0.1:8089</server_uri>``
`` <session_key>123102983109283019283</session_key>``
`` <checkpoint_dir>/opt/splunk/var/lib/splunk/modinputs</checkpoint_dir>``
`` <item name="myScheme">``
`` <param name="param1">value1</param>``
`` <param_list name="param2">``
`` <value>value2</value>``
`` <value>value3</value>``
`` <value>value4</value>``
`` </param_list>``
`` </item>``
``</items>``
:param stream: ``Stream`` containing XML to parse.
:return definition: A ``ValidationDefinition`` object.
"""
definition = ValidationDefinition()
# parse XML from the stream, then get the root node
root = ET.parse(stream).getroot()
for node in root:
# lone item node
if node.tag == "item":
# name from item node
definition.metadata["name"] = node.get("name")
definition.parameters = parse_xml_data(node, "")
else:
# Store anything else in metadata
definition.metadata[node.tag] = node.text
return definition
package/bin/splunklib/ordereddict.py
Executable
+128
View File
@@ -0,0 +1,128 @@
# Copyright (c) 2009 Raymond Hettinger
#
# Permission is hereby granted, free of charge, to any person
# obtaining a copy of this software and associated documentation files
# (the "Software"), to deal in the Software without restriction,
# including without limitation the rights to use, copy, modify, merge,
# publish, distribute, sublicense, and/or sell copies of the Software,
# and to permit persons to whom the Software is furnished to do so,
# subject to the following conditions:
#
# The above copyright notice and this permission notice shall be
# included in all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
# OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
# HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
# WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
# OTHER DEALINGS IN THE SOFTWARE.
from UserDict import DictMixin
class OrderedDict(dict, DictMixin):
def __init__(self, *args, **kwds):
if len(args) > 1:
raise TypeError('expected at most 1 arguments, got %d' % len(args))
try:
self.__end
except AttributeError:
self.clear()
self.update(*args, **kwds)
def clear(self):
self.__end = end = []
end += [None, end, end] # sentinel node for doubly linked list
self.__map = {} # key --> [key, prev, next]
dict.clear(self)
def __setitem__(self, key, value):
if key not in self:
end = self.__end
curr = end[1]
curr[2] = end[1] = self.__map[key] = [key, curr, end]
dict.__setitem__(self, key, value)
def __delitem__(self, key):
dict.__delitem__(self, key)
key, prev, next = self.__map.pop(key)
prev[2] = next
next[1] = prev
def __iter__(self):
end = self.__end
curr = end[2]
while curr is not end:
yield curr[0]
curr = curr[2]
def __reversed__(self):
end = self.__end
curr = end[1]
while curr is not end:
yield curr[0]
curr = curr[1]
def popitem(self, last=True):
if not self:
raise KeyError('dictionary is empty')
if last:
key = reversed(self).next()
else:
key = iter(self).next()
value = self.pop(key)
return key, value
def __reduce__(self):
items = [[k, self[k]] for k in self]
tmp = self.__map, self.__end
del self.__map, self.__end
inst_dict = vars(self).copy()
self.__map, self.__end = tmp
if inst_dict:
return (self.__class__, (items,), inst_dict)
return self.__class__, (items,)
def keys(self):
return list(self)
setdefault = DictMixin.setdefault
update = DictMixin.update
pop = DictMixin.pop
values = DictMixin.values
items = DictMixin.items
iterkeys = DictMixin.iterkeys
itervalues = DictMixin.itervalues
iteritems = DictMixin.iteritems
def __repr__(self):
if not self:
return '%s()' % (self.__class__.__name__,)
return '%s(%r)' % (self.__class__.__name__, self.items())
def copy(self):
return self.__class__(self)
@classmethod
def fromkeys(cls, iterable, value=None):
d = cls()
for key in iterable:
d[key] = value
return d
def __eq__(self, other):
if isinstance(other, OrderedDict):
if len(self) != len(other):
return False
for p, q in zip(self.items(), other.items()):
if p != q:
return False
return True
return dict.__eq__(self, other)
def __ne__(self, other):
return not self == other
package/bin/splunklib/results.py
Executable
+288
View File
@@ -0,0 +1,288 @@
# Copyright 2011-2015 Splunk, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"): you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
"""The **splunklib.results** module provides a streaming XML reader for Splunk
search results.
Splunk search results can be returned in a variety of formats including XML,
JSON, and CSV. To make it easier to stream search results in XML format, they
are returned as a stream of XML *fragments*, not as a single XML document. This
module supports incrementally reading one result record at a time from such a
result stream. This module also provides a friendly iterator-based interface for
accessing search results while avoiding buffering the result set, which can be
very large.
To use the reader, instantiate :class:`ResultsReader` on a search result stream
as follows:::
reader = ResultsReader(result_stream)
for item in reader:
print(item)
print "Results are a preview: %s" % reader.is_preview
"""
try:
import xml.etree.cElementTree as et
except:
import xml.etree.ElementTree as et
try:
from collections import OrderedDict # must be python 2.7
except ImportError:
from .ordereddict import OrderedDict
try:
from cStringIO import StringIO
except:
from StringIO import StringIO
__all__ = [
"ResultsReader",
"Message"
]
class Message(object):
"""This class represents informational messages that Splunk interleaves in the results stream.
``Message`` takes two arguments: a string giving the message type (e.g., "DEBUG"), and
a string giving the message itself.
**Example**::
m = Message("DEBUG", "There's something in that variable...")
"""
def __init__(self, type_, message):
self.type = type_
self.message = message
def __repr__(self):
return "%s: %s" % (self.type, self.message)
def __eq__(self, other):
return (self.type, self.message) == (other.type, other.message)
def __hash__(self):
return hash((self.type, self.message))
class _ConcatenatedStream(object):
"""Lazily concatenate zero or more streams into a stream.
As you read from the concatenated stream, you get characters from
each stream passed to ``_ConcatenatedStream``, in order.
**Example**::
from StringIO import StringIO
s = _ConcatenatedStream(StringIO("abc"), StringIO("def"))
assert s.read() == "abcdef"
"""
def __init__(self, *streams):
self.streams = list(streams)
def read(self, n=None):
"""Read at most *n* characters from this stream.
If *n* is ``None``, return all available characters.
"""
response = ""
while len(self.streams) > 0 and (n is None or n > 0):
txt = self.streams[0].read(n)
response += txt
if n is not None:
n -= len(txt)
if n > 0 or n is None:
del self.streams[0]
return response
class _XMLDTDFilter(object):
"""Lazily remove all XML DTDs from a stream.
All substrings matching the regular expression <?[^>]*> are
removed in their entirety from the stream. No regular expressions
are used, however, so everything still streams properly.
**Example**::
from StringIO import StringIO
s = _XMLDTDFilter("<?xml abcd><element><?xml ...></element>")
assert s.read() == "<element></element>"
"""
def __init__(self, stream):
self.stream = stream
def read(self, n=None):
"""Read at most *n* characters from this stream.
If *n* is ``None``, return all available characters.
"""
response = ""
while n is None or n > 0:
c = self.stream.read(1)
if c == "":
break
elif c == "<":
c += self.stream.read(1)
if c == "<?":
while True:
q = self.stream.read(1)
if q == ">":
break
else:
response += c
if n is not None:
n -= len(c)
else:
response += c
if n is not None:
n -= 1
return response
class ResultsReader(object):
"""This class returns dictionaries and Splunk messages from an XML results
stream.
``ResultsReader`` is iterable, and returns a ``dict`` for results, or a
:class:`Message` object for Splunk messages. This class has one field,
``is_preview``, which is ``True`` when the results are a preview from a
running search, or ``False`` when the results are from a completed search.
This function has no network activity other than what is implicit in the
stream it operates on.
:param `stream`: The stream to read from (any object that supports
``.read()``).
**Example**::
import results
response = ... # the body of an HTTP response
reader = results.ResultsReader(response)
for result in reader:
if isinstance(result, dict):
print "Result: %s" % result
elif isinstance(result, results.Message):
print "Message: %s" % result
print "is_preview = %s " % reader.is_preview
"""
# Be sure to update the docstrings of client.Jobs.oneshot,
# client.Job.results_preview and client.Job.results to match any
# changes made to ResultsReader.
#
# This wouldn't be a class, just the _parse_results function below,
# except that you cannot get the current generator inside the
# function creating that generator. Thus it's all wrapped up for
# the sake of one field.
def __init__(self, stream):
# The search/jobs/exports endpoint, when run with
# earliest_time=rt and latest_time=rt streams a sequence of
# XML documents, each containing a result, as opposed to one
# results element containing lots of results. Python's XML
# parsers are broken, and instead of reading one full document
# and returning the stream that follows untouched, they
# destroy the stream and throw an error. To get around this,
# we remove all the DTD definitions inline, then wrap the
# fragments in a fiction <doc> element to make the parser happy.
stream = _XMLDTDFilter(stream)
stream = _ConcatenatedStream(StringIO("<doc>"), stream, StringIO("</doc>"))
self.is_preview = None
self._gen = self._parse_results(stream)
def __iter__(self):
return self
def next(self):
return self._gen.next()
def _parse_results(self, stream):
"""Parse results and messages out of *stream*."""
result = None
values = None
try:
for event, elem in et.iterparse(stream, events=('start', 'end')):
if elem.tag == 'results' and event == 'start':
# The wrapper element is a <results preview="0|1">. We
# don't care about it except to tell is whether these
# are preview results, or the final results from the
# search.
is_preview = elem.attrib['preview'] == '1'
self.is_preview = is_preview
if elem.tag == 'result':
if event == 'start':
result = OrderedDict()
elif event == 'end':
yield result
result = None
elem.clear()
elif elem.tag == 'field' and result is not None:
# We need the 'result is not None' check because
# 'field' is also the element name in the <meta>
# header that gives field order, which is not what we
# want at all.
if event == 'start':
values = []
elif event == 'end':
field_name = elem.attrib['k'].encode('utf8')
if len(values) == 1:
result[field_name] = values[0]
else:
result[field_name] = values
# Calling .clear() is necessary to let the
# element be garbage collected. Otherwise
# arbitrarily large results sets will use
# arbitrarily large memory intead of
# streaming.
elem.clear()
elif elem.tag in ('text', 'v') and event == 'end':
try:
text = "".join(elem.itertext())
except AttributeError:
# Assume we're running in Python < 2.7, before itertext() was added
# So we'll define it here
def __itertext(self):
tag = self.tag
if not isinstance(tag, basestring) and tag is not None:
return
if self.text:
yield self.text
for e in self:
for s in __itertext(e):
yield s
if e.tail:
yield e.tail
text = "".join(__itertext(elem))
values.append(text.encode('utf8'))
elem.clear()
elif elem.tag == 'msg':
if event == 'start':
msg_type = elem.attrib['type']
elif event == 'end':
text = elem.text if elem.text is not None else ""
yield Message(msg_type, text.encode('utf8'))
elem.clear()
except SyntaxError as pe:
# This is here to handle the same incorrect return from
# splunk that is described in __init__.
if 'no element found' in pe.msg:
return
else:
raise
package/bin/splunklib/searchcommands/__init__.py
Executable
+155
View File
@@ -0,0 +1,155 @@
# coding=utf-8
#
# Copyright © 2011-2015 Splunk, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"): you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
"""
.. topic:: Design Notes
1. Commands are constrained to this ABNF grammar::
command = command-name *[wsp option] *[wsp [dquote] field-name [dquote]]
command-name = alpha *( alpha / digit )
option = option-name [wsp] "=" [wsp] option-value
option-name = alpha *( alpha / digit / "_" )
option-value = word / quoted-string
word = 1*( %01-%08 / %0B / %0C / %0E-1F / %21 / %23-%FF ) ; Any character but DQUOTE and WSP
quoted-string = dquote *( word / wsp / "\" dquote / dquote dquote ) dquote
field-name = ( "_" / alpha ) *( alpha / digit / "_" / "." / "-" )
It does not show that :code:`field-name` values may be comma-separated. This is because Splunk strips commas from
the command line. A search command will never see them.
2. Search commands targeting versions of Splunk prior to 6.3 must be statically configured as follows:
.. code-block:: text
:linenos:
[command_name]
filename = command_name.py
supports_getinfo = true
supports_rawargs = true
No other static configuration is required or expected and may interfere with command execution.
3. Commands support dynamic probing for settings.
Splunk probes for settings dynamically when :code:`supports_getinfo=true`.
You must add this line to the commands.conf stanza for each of your search
commands.
4. Commands do not support parsed arguments on the command line.
Splunk parses arguments when :code:`supports_rawargs=false`. The
:code:`SearchCommand` class sets this value unconditionally. You cannot
override it.
**Rationale**
Splunk parses arguments by stripping quotes, nothing more. This may be useful
in some cases, but doesn't work well with our chosen grammar.
5. Commands consume input headers.
An input header is provided by Splunk when :code:`enableheader=true`. The
:class:`SearchCommand` class sets this value unconditionally. You cannot
override it.
6. Commands produce an output messages header.
Splunk expects a command to produce an output messages header when
:code:`outputheader=true`. The :class:`SearchCommand` class sets this value
unconditionally. You cannot override it.
7. Commands support multi-value fields.
Multi-value fields are provided and consumed by Splunk when
:code:`supports_multivalue=true`. This value is fixed. You cannot override
it.
8. This module represents all fields on the output stream in multi-value
format.
Splunk recognizes two kinds of data: :code:`value` and :code:`list(value)`.
The multi-value format represents these data in field pairs. Given field
:code:`name` the multi-value format calls for the creation of this pair of
fields.
================= =========================================================
Field name Field data
================= =========================================================
:code:`name` Value or text from which a list of values was derived.
:code:`__mv_name` Empty, if :code:`field` represents a :code:`value`;
otherwise, an encoded :code:`list(value)`. Values in the
list are wrapped in dollar signs ($) and separated by
semi-colons (;). Dollar signs ($) within a value are
represented by a pair of dollar signs ($$).
================= =========================================================
Serializing data in this format enables streaming and reduces a command's
memory footprint at the cost of one extra byte of data per field per record
and a small amount of extra processing time by the next command in the
pipeline.
9. A :class:`ReportingCommand` must override :meth:`~ReportingCommand.reduce`
and may override :meth:`~ReportingCommand.map`. Map/reduce commands on the
Splunk processing pipeline are distinguished as this example illustrates.
**Splunk command**
.. code-block:: text
sum total=total_date_hour date_hour
**Map command line**
.. code-block:: text
sum __GETINFO__ __map__ total=total_date_hour date_hour
sum __EXECUTE__ __map__ total=total_date_hour date_hour
**Reduce command line**
.. code-block:: text
sum __GETINFO__ total=total_date_hour date_hour
sum __EXECUTE__ total=total_date_hour date_hour
The :code:`__map__` argument is introduced by
:meth:`ReportingCommand._execute`. Search command authors cannot influence
the contents of the command line in this release.
.. topic:: References
1. `Search command style guide <http://docs.splunk.com/Documentation/Splunk/6.0/Search/Searchcommandstyleguide>`_
2. `Commands.conf.spec <http://docs.splunk.com/Documentation/Splunk/5.0.5/Admin/Commandsconf>`_
"""
from __future__ import absolute_import, division, print_function, unicode_literals
from .environment import *
from .decorators import *
from .validators import *
from .generating_command import GeneratingCommand
from .streaming_command import StreamingCommand
from .eventing_command import EventingCommand
from .reporting_command import ReportingCommand
from .external_search_command import execute, ExternalSearchCommand
from .search_command import dispatch, SearchMetric
package/bin/splunklib/searchcommands/decorators.py
Executable
+447
View File
@@ -0,0 +1,447 @@
# coding=utf-8
#
# Copyright © 2011-2015 Splunk, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"): you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from __future__ import absolute_import, division, print_function, unicode_literals
try:
from collections import OrderedDict # must be python 2.7
except ImportError:
from ..ordereddict import OrderedDict
from inspect import getmembers, isclass, isfunction
from itertools import imap
from .internals import ConfigurationSettingsType, json_encode_string
from .validators import OptionName
class Configuration(object):
""" Defines the configuration settings for a search command.
Documents, validates, and ensures that only relevant configuration settings are applied. Adds a :code:`name` class
variable to search command classes that don't have one. The :code:`name` is derived from the name of the class.
By convention command class names end with the word "Command". To derive :code:`name` the word "Command" is removed
from the end of the class name and then converted to lower case for conformance with the `Search command style guide
<http://docs.splunk.com/Documentation/Splunk/latest/Search/Searchcommandstyleguide>`_
"""
def __init__(self, o=None, **kwargs):
#
# The o argument enables the configuration decorator to be used with or without parentheses. For example, it
# enables you to write code that looks like this:
#
# @Configuration
# class Foo(SearchCommand):
# ...
#
# @Configuration()
# class Bar(SearchCommand):
# ...
#
# Without the o argument, the Python compiler will complain about the first form. With the o argument, both
# forms work. The first form provides a value for o: Foo. The second form does does not provide a value for o.
# The class or method decorated is not passed to the constructor. A value of None is passed instead.
#
self.settings = kwargs
def __call__(self, o):
if isfunction(o):
# We must wait to finalize configuration as the class containing this function is under construction
# at the time this call to decorate a member function. This will be handled in the call to
# o.ConfigurationSettings.fix_up(o) in the elif clause of this code block.
o._settings = self.settings
elif isclass(o):
# Set command name
name = o.__name__
if name.endswith(b'Command'):
name = name[:-len(b'Command')]
o.name = unicode(name.lower())
# Construct ConfigurationSettings instance for the command class
o.ConfigurationSettings = ConfigurationSettingsType(
module=o.__module__ + b'.' + o.__name__,
name=b'ConfigurationSettings',
bases=(o.ConfigurationSettings,))
ConfigurationSetting.fix_up(o.ConfigurationSettings, self.settings)
o.ConfigurationSettings.fix_up(o)
Option.fix_up(o)
else:
raise TypeError('Incorrect usage: Configuration decorator applied to {0}'.format(type(o), o.__name__))
return o
class ConfigurationSetting(property):
""" Generates a :class:`property` representing the named configuration setting
This is a convenience function designed to reduce the amount of boiler-plate code you must write; most notably for
property setters.
:param name: Configuration setting name.
:type name: str or unicode
:param doc: A documentation string.
:type doc: bytes, unicode or NoneType
:param readonly: If true, specifies that the configuration setting is fixed.
:type name: bool or NoneType
:param value: Configuration setting value.
:return: A :class:`property` instance representing the configuration setting.
:rtype: property
"""
def __init__(self, fget=None, fset=None, fdel=None, doc=None, name=None, readonly=None, value=None):
property.__init__(self, fget=fget, fset=fset, fdel=fdel, doc=doc)
self._readonly = readonly
self._value = value
self._name = name
def __call__(self, function):
return self.getter(function)
def deleter(self, function):
return self._copy_extra_attributes(property.deleter(self, function))
def getter(self, function):
return self._copy_extra_attributes(property.getter(self, function))
def setter(self, function):
return self._copy_extra_attributes(property.setter(self, function))
@staticmethod
def fix_up(cls, values):
is_configuration_setting = lambda attribute: isinstance(attribute, ConfigurationSetting)
definitions = getmembers(cls, is_configuration_setting)
i = 0
for name, setting in definitions:
if setting._name is None:
setting._name = name = unicode(name)
else:
name = setting._name
validate, specification = setting._get_specification()
backing_field_name = '_' + name
if setting.fget is None and setting.fset is None and setting.fdel is None:
value = setting._value
if setting._readonly or value is not None:
validate(specification, name, value)
def fget(bfn, value):
return lambda this: getattr(this, bfn, value)
setting = setting.getter(fget(backing_field_name, value))
if not setting._readonly:
def fset(bfn, validate, specification, name):
return lambda this, value: setattr(this, bfn, validate(specification, name, value))
setting = setting.setter(fset(backing_field_name, validate, specification, name))
setattr(cls, name, setting)
def is_supported_by_protocol(supporting_protocols):
def is_supported_by_protocol(version):
return version in supporting_protocols
return is_supported_by_protocol
del setting._name, setting._value, setting._readonly
setting.is_supported_by_protocol = is_supported_by_protocol(specification.supporting_protocols)
setting.supporting_protocols = specification.supporting_protocols
setting.backing_field_name = backing_field_name
definitions[i] = setting
setting.name = name
i += 1
try:
value = values[name]
except KeyError:
continue
if setting.fset is None:
raise ValueError('The value of configuration setting {} is fixed'.format(name))
setattr(cls, backing_field_name, validate(specification, name, value))
del values[name]
if len(values) > 0:
settings = sorted(list(values.iteritems()))
settings = imap(lambda (n, v): '{}={}'.format(n, repr(v)), settings)
raise AttributeError('Inapplicable configuration settings: ' + ', '.join(settings))
cls.configuration_setting_definitions = definitions
def _copy_extra_attributes(self, other):
other._readonly = self._readonly
other._value = self._value
other._name = self._name
return other
def _get_specification(self):
name = self._name
try:
specification = ConfigurationSettingsType.specification_matrix[name]
except KeyError:
raise AttributeError('Unknown configuration setting: {}={}'.format(name, repr(self._value)))
return ConfigurationSettingsType.validate_configuration_setting, specification
class Option(property):
""" Represents a search command option.
Required options must be specified on the search command line.
**Example:**
Short form (recommended). When you are satisfied with built-in or custom validation behaviors.
.. code-block:: python
:linenos:
from splunklib.searchcommands.decorators import Option
from splunklib.searchcommands.validators import Fieldname
total = Option(
doc=''' **Syntax:** **total=***<fieldname>*
**Description:** Name of the field that will hold the computed
sum''',
require=True, validate=Fieldname())
**Example:**
Long form. Useful when you wish to manage the option value and its deleter/getter/setter side-effects yourself. You
must provide a getter and a setter. If your :code:`Option` requires `destruction <https://docs.python.org/2/reference/datamodel.html#object.__del__>`_ you must
also provide a deleter. You must be prepared to accept a value of :const:`None` which indicates that your
:code:`Option` is unset.
.. code-block:: python
:linenos:
from splunklib.searchcommands import Option
@Option()
def logging_configuration(self):
\""" **Syntax:** logging_configuration=<path>
**Description:** Loads an alternative logging configuration file for a command invocation. The logging
configuration file must be in Python ConfigParser-format. The *<path>* name and all path names specified in
configuration are relative to the app root directory.
\"""
return self._logging_configuration
@logging_configuration.setter
def logging_configuration(self, value):
if value is not None
logging.configure(value)
self._logging_configuration = value
def __init__(self)
self._logging_configuration = None
"""
def __init__(self, fget=None, fset=None, fdel=None, doc=None, name=None, default=None, require=None, validate=None):
property.__init__(self, fget, fset, fdel, doc)
self.name = name
self.default = default
self.validate = validate
self.require = bool(require)
def __call__(self, function):
return self.getter(function)
# region Methods
def deleter(self, function):
return self._copy_extra_attributes(property.deleter(self, function))
def getter(self, function):
return self._copy_extra_attributes(property.getter(self, function))
def setter(self, function):
return self._copy_extra_attributes(property.setter(self, function))
@classmethod
def fix_up(cls, command_class):
is_option = lambda attribute: isinstance(attribute, Option)
definitions = getmembers(command_class, is_option)
validate_option_name = OptionName()
i = 0
for name, option in definitions:
if option.name is None:
option.name = name # no validation required
else:
validate_option_name(option.name)
if option.fget is None and option.fset is None and option.fdel is None:
backing_field_name = '_' + name
def fget(bfn):
return lambda this: getattr(this, bfn, None)
option = option.getter(fget(backing_field_name))
def fset(bfn, validate):
if validate is None:
return lambda this, value: setattr(this, bfn, value)
return lambda this, value: setattr(this, bfn, validate(value))
option = option.setter(fset(backing_field_name, option.validate))
setattr(command_class, name, option)
elif option.validate is not None:
def fset(function, validate):
return lambda this, value: function(this, validate(value))
option = option.setter(fset(option.fset, option.validate))
setattr(command_class, name, option)
definitions[i] = name, option
i += 1
command_class.option_definitions = definitions
def _copy_extra_attributes(self, other):
other.name = self.name
other.default = self.default
other.require = self.require
other.validate = self.validate
return other
# endregion
# region Types
class Item(object):
""" Presents an instance/class view over a search command `Option`.
This class is used by SearchCommand.process to parse and report on option values.
"""
def __init__(self, command, option):
self._command = command
self._option = option
self._is_set = False
validator = self.validator
self._format = unicode if validator is None else validator.format
def __repr__(self):
return '(' + repr(self.name) + ', ' + repr(self._format(self.value)) + ')'
def __str__(self):
value = self.value
value = 'None' if value is None else json_encode_string(self._format(value))
return self.name + '=' + value
# region Properties
@property
def is_required(self):
return bool(self._option.require)
@property
def is_set(self):
""" Indicates whether an option value was provided as argument.
"""
return self._is_set
@property
def name(self):
return self._option.name
@property
def validator(self):
return self._option.validate
@property
def value(self):
return self._option.__get__(self._command)
@value.setter
def value(self, value):
self._option.__set__(self._command, value)
self._is_set = True
# endregion
# region Methods
def reset(self):
self._option.__set__(self._command, self._option.default)
self._is_set = False
pass
# endregion
class View(OrderedDict):
""" Presents an ordered dictionary view of the set of :class:`Option` arguments to a search command.
This class is used by SearchCommand.process to parse and report on option values.
"""
def __init__(self, command):
definitions = type(command).option_definitions
item_class = Option.Item
OrderedDict.__init__(self, imap(lambda (name, option): (option.name, item_class(command, option)), definitions))
def __repr__(self):
text = 'Option.View([' + ','.join(imap(lambda item: repr(item), self.itervalues())) + '])'
return text
def __str__(self):
text = ' '.join([str(item) for item in self.itervalues() if item.is_set])
return text
# region Methods
def get_missing(self):
missing = [item.name for item in self.itervalues() if item.is_required and not item.is_set]
return missing if len(missing) > 0 else None
def reset(self):
for value in self.itervalues():
value.reset()
pass
# endregion
pass
# endregion
__all__ = ['Configuration', 'Option']
package/bin/splunklib/searchcommands/environment.py
Executable
+122
View File
@@ -0,0 +1,122 @@
# coding=utf-8
#
# Copyright © 2011-2015 Splunk, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"): you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from __future__ import absolute_import, division, print_function, unicode_literals
from logging import getLogger, root, StreamHandler
from logging.config import fileConfig
from os import chdir, environ, getcwdu, path
import sys
def configure_logging(logger_name, filename=None):
""" Configure logging and return the named logger and the location of the logging configuration file loaded.
This function expects a Splunk app directory structure::
<app-root>
bin
...
default
...
local
...
This function looks for a logging configuration file at each of these locations, loading the first, if any,
logging configuration file that it finds::
local/{name}.logging.conf
default/{name}.logging.conf
local/logging.conf
default/logging.conf
The current working directory is set to *<app-root>* before the logging configuration file is loaded. Hence, paths
in the logging configuration file are relative to *<app-root>*. The current directory is reset before return.
You may short circuit the search for a logging configuration file by providing an alternative file location in
`path`. Logging configuration files must be in `ConfigParser format`_.
#Arguments:
:param logger_name: Logger name
:type logger_name: bytes, unicode
:param filename: Location of an alternative logging configuration file or `None`.
:type filename: bytes, unicode or NoneType
:returns: The named logger and the location of the logging configuration file loaded.
:rtype: tuple
.. _ConfigParser format: https://docs.python.org/2/library/logging.config.html#configuration-file-format
"""
if filename is None:
if logger_name is None:
probing_paths = [path.join('local', 'logging.conf'), path.join('default', 'logging.conf')]
else:
probing_paths = [
path.join('local', logger_name + '.logging.conf'),
path.join('default', logger_name + '.logging.conf'),
path.join('local', 'logging.conf'),
path.join('default', 'logging.conf')]
for relative_path in probing_paths:
configuration_file = path.join(app_root, relative_path)
if path.exists(configuration_file):
filename = configuration_file
break
elif not path.isabs(filename):
found = False
for conf in 'local', 'default':
configuration_file = path.join(app_root, conf, filename)
if path.exists(configuration_file):
filename = configuration_file
found = True
break
if not found:
raise ValueError('Logging configuration file "{}" not found in local or default directory'.format(filename))
elif not path.exists(filename):
raise ValueError('Logging configuration file "{}" not found'.format(filename))
if filename is not None:
global _current_logging_configuration_file
filename = path.realpath(filename)
if filename != _current_logging_configuration_file:
working_directory = getcwdu()
chdir(app_root)
try:
fileConfig(filename, {'SPLUNK_HOME': splunk_home})
finally:
chdir(working_directory)
_current_logging_configuration_file = filename
if len(root.handlers) == 0:
root.addHandler(StreamHandler())
return None if logger_name is None else getLogger(logger_name), filename
_current_logging_configuration_file = None
splunk_home = path.abspath(path.join(getcwdu(), environ.get('SPLUNK_HOME', '')))
app_file = getattr(sys.modules['__main__'], '__file__', sys.executable)
app_root = path.dirname(path.abspath(path.dirname(app_file)))
splunklib_logger, logging_configuration = configure_logging('splunklib')
__all__ = ['app_file', 'app_root', 'logging_configuration', 'splunk_home', 'splunklib_logger']
package/bin/splunklib/searchcommands/eventing_command.py
Executable
+142
View File
@@ -0,0 +1,142 @@
# coding=utf-8
#
# Copyright 2011-2015 Splunk, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"): you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from __future__ import absolute_import, division, print_function, unicode_literals
from itertools import imap
from .decorators import ConfigurationSetting
from .search_command import SearchCommand
class EventingCommand(SearchCommand):
""" Applies a transformation to search results as they travel through the events pipeline.
Eventing commands typically filter, group, order, and/or or augment event records. Examples of eventing commands
from Splunk's built-in command set include sort_, dedup_, and cluster_. Each execution of an eventing command
should produce a set of event records that is independently usable by downstream processors.
.. _sort: http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Sort
.. _dedup: http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Dedup
.. _cluster: http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Cluster
EventingCommand configuration
==============================
You can configure your command for operation under Search Command Protocol (SCP) version 1 or 2. SCP 2 requires
Splunk 6.3 or later.
"""
# region Methods
def transform(self, records):
""" Generator function that processes and yields event records to the Splunk events pipeline.
You must override this method.
"""
raise NotImplementedError('EventingCommand.transform(self, records)')
def _execute(self, ifile, process):
SearchCommand._execute(self, ifile, self.transform)
# endregion
class ConfigurationSettings(SearchCommand.ConfigurationSettings):
""" Represents the configuration settings that apply to a :class:`EventingCommand`.
"""
# region SCP v1/v2 properties
required_fields = ConfigurationSetting(doc='''
List of required fields for this search which back-propagates to the generating search.
Setting this value enables selected fields mode under SCP 2. Under SCP 1 you must also specify
:code:`clear_required_fields=True` to enable selected fields mode. To explicitly select all fields,
specify a value of :const:`['*']`. No error is generated if a specified field is missing.
Default: :const:`None`, which implicitly selects all fields.
''')
# endregion
# region SCP v1 properties
clear_required_fields = ConfigurationSetting(doc='''
:const:`True`, if required_fields represent the *only* fields required.
If :const:`False`, required_fields are additive to any fields that may be required by subsequent commands.
In most cases, :const:`False` is appropriate for eventing commands.
Default: :const:`False`
''')
retainsevents = ConfigurationSetting(readonly=True, value=True, doc='''
:const:`True`, if the command retains events the way the sort/dedup/cluster commands do.
If :const:`False`, the command transforms events the way the stats command does.
Fixed: :const:`True`
''')
# endregion
# region SCP v2 properties
maxinputs = ConfigurationSetting(doc='''
Specifies the maximum number of events that can be passed to the command for each invocation.
This limit cannot exceed the value of `maxresultrows` as defined in limits.conf_. Under SCP 1 you must
specify this value in commands.conf_.
Default: The value of `maxresultrows`.
Supported by: SCP 2
.. _limits.conf: http://docs.splunk.com/Documentation/Splunk/latest/admin/Limitsconf
''')
type = ConfigurationSetting(readonly=True, value='eventing', doc='''
Command type
Fixed: :const:`'eventing'`.
Supported by: SCP 2
''')
# endregion
# region Methods
@classmethod
def fix_up(cls, command):
""" Verifies :code:`command` class structure.
"""
if command.transform == EventingCommand.transform:
raise AttributeError('No EventingCommand.transform override')
SearchCommand.ConfigurationSettings.fix_up(command)
def iteritems(self):
iteritems = SearchCommand.ConfigurationSettings.iteritems(self)
return imap(lambda (name, value): (name, 'events' if name == 'type' else value), iteritems)
# endregion
package/bin/splunklib/searchcommands/external_search_command.py
Executable
+227
View File
@@ -0,0 +1,227 @@
# coding=utf-8
#
# Copyright 2011-2015 Splunk, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"): you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from __future__ import absolute_import, division, print_function, unicode_literals
from logging import getLogger
import os
import sys
import traceback
if sys.platform == 'win32':
from signal import signal, CTRL_BREAK_EVENT, SIGBREAK, SIGINT, SIGTERM
from subprocess import Popen
import atexit
from . import splunklib_logger as logger
# P1 [ ] TODO: Add ExternalSearchCommand class documentation
class ExternalSearchCommand(object):
"""
"""
def __init__(self, path, argv=None, environ=None):
if not isinstance(path, (bytes, unicode)):
raise ValueError('Expected a string value for path, not {}'.format(repr(path)))
self._logger = getLogger(self.__class__.__name__)
self._path = unicode(path)
self._argv = None
self._environ = None
self.argv = argv
self.environ = environ
# region Properties
@property
def argv(self):
return getattr(self, '_argv')
@argv.setter
def argv(self, value):
if not (value is None or isinstance(value, (list, tuple))):
raise ValueError('Expected a list, tuple or value of None for argv, not {}'.format(repr(value)))
self._argv = value
@property
def environ(self):
return getattr(self, '_environ')
@environ.setter
def environ(self, value):
if not (value is None or isinstance(value, dict)):
raise ValueError('Expected a dictionary value for environ, not {}'.format(repr(value)))
self._environ = value
@property
def logger(self):
return self._logger
@property
def path(self):
return self._path
# endregion
# region Methods
def execute(self):
# noinspection PyBroadException
try:
if self._argv is None:
self._argv = os.path.splitext(os.path.basename(self._path))[0]
self._execute(self._path, self._argv, self._environ)
except:
error_type, error, tb = sys.exc_info()
message = 'Command execution failed: ' + unicode(error)
self._logger.error(message + '\nTraceback:\n' + ''.join(traceback.format_tb(tb)))
sys.exit(1)
if sys.platform == 'win32':
@staticmethod
def _execute(path, argv=None, environ=None):
""" Executes an external search command.
:param path: Path to the external search command.
:type path: unicode
:param argv: Argument list.
:type argv: list or tuple
The arguments to the child process should start with the name of the command being run, but this is not
enforced. A value of :const:`None` specifies that the base name of path name :param:`path` should be used.
:param environ: A mapping which is used to define the environment variables for the new process.
:type environ: dict or None.
This mapping is used instead of the current process’s environment. A value of :const:`None` specifies that
the :data:`os.environ` mapping should be used.
:return: None
"""
search_path = os.getenv('PATH') if environ is None else environ.get('PATH')
found = ExternalSearchCommand._search_path(path, search_path)
if found is None:
raise ValueError('Cannot find command on path: {}'.format(path))
path = found
logger.debug('starting command="%s", arguments=%s', path, argv)
def terminate(signal_number, frame):
sys.exit('External search command is terminating on receipt of signal={}.'.format(signal_number))
def terminate_child():
if p.pid is not None and p.returncode is None:
logger.debug('terminating command="%s", arguments=%d, pid=%d', path, argv, p.pid)
os.kill(p.pid, CTRL_BREAK_EVENT)
p = Popen(argv, executable=path, env=environ, stdin=sys.stdin, stdout=sys.stdout, stderr=sys.stderr)
atexit.register(terminate_child)
signal(SIGBREAK, terminate)
signal(SIGINT, terminate)
signal(SIGTERM, terminate)
logger.debug('started command="%s", arguments=%s, pid=%d', path, argv, p.pid)
p.wait()
logger.debug('finished command="%s", arguments=%s, pid=%d, returncode=%d', path, argv, p.pid, p.returncode)
if p.returncode != 0:
sys.exit(p.returncode)
@staticmethod
def _search_path(executable, paths):
""" Locates an executable program file.
:param executable: The name of the executable program to locate.
:type executable: unicode
:param paths: A list of one or more directory paths where executable programs are located.
:type paths: unicode
:return:
:rtype: Path to the executable program located or :const:`None`.
"""
directory, filename = os.path.split(executable)
extension = os.path.splitext(filename)[1].upper()
executable_extensions = ExternalSearchCommand._executable_extensions
if directory:
if len(extension) and extension in executable_extensions:
return None
for extension in executable_extensions:
path = executable + extension
if os.path.isfile(path):
return path
return None
if not paths:
return None
directories = [directory for directory in paths.split(';') if len(directory)]
if len(directories) == 0:
return None
if len(extension) and extension in executable_extensions:
for directory in directories:
path = os.path.join(directory, executable)
if os.path.isfile(path):
return path
return None
for directory in directories:
path_without_extension = os.path.join(directory, executable)
for extension in executable_extensions:
path = path_without_extension + extension
if os.path.isfile(path):
return path
return None
_executable_extensions = ('.COM', '.EXE')
else:
@staticmethod
def _execute(path, argv, environ):
if environ is None:
os.execvp(path, argv)
else:
os.execvpe(path, argv, environ)
return
# endregion
def execute(path, argv=None, environ=None, command_class=ExternalSearchCommand):
"""
:param path:
:type path: basestring
:param argv:
:type: argv: list, tuple, or None
:param environ:
:type environ: dict
:param command_class: External search command class to instantiate and execute.
:type command_class: type
:return:
:rtype: None
"""
assert issubclass(command_class, ExternalSearchCommand)
command_class(path, argv, environ).execute()
package/bin/splunklib/searchcommands/generating_command.py
Executable
+320
View File
@@ -0,0 +1,320 @@
# coding=utf-8
#
# Copyright © 2011-2015 Splunk, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"): you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from __future__ import absolute_import, division, print_function, unicode_literals
from .decorators import ConfigurationSetting
from .search_command import SearchCommand
from itertools import imap, ifilter
# P1 [O] TODO: Discuss generates_timeorder in the class-level documentation for GeneratingCommand
class GeneratingCommand(SearchCommand):
""" Generates events based on command arguments.
Generating commands receive no input and must be the first command on a pipeline. There are three pipelines:
streams, events, and reports. The streams pipeline generates or processes time-ordered event records on an
indexer or search head.
Streaming commands filter, modify, or augment event records and can be applied to subsets of index data in a
parallel manner. An example of a streaming command from Splunk's built-in command set is rex_ which extracts and
adds fields to event records at search time. Records that pass through the streams pipeline move on to the events
pipeline.
The events pipeline generates or processes records on a search head. Eventing commands typically filter, group,
order, or augment event records. Examples of eventing commands from Splunk's built-in command set include sort_,
dedup_, and cluster_. Each execution of an eventing command should produce a set of event records that is
independently usable by downstream processors. Records that pass through the events pipeline move on to the reports
pipeline.
The reports pipeline also runs on a search head, but yields data structures for presentation, not event records.
Examples of streaming from Splunk's built-in command set include chart_, stats_, and contingency_.
GeneratingCommand configuration
===============================
Configure your generating command based on the pipeline that it targets. How you configure your command depends on
the Search Command Protocol (SCP) version.
+----------+-------------------------------------+--------------------------------------------+
| Pipeline | SCP 1 | SCP 2 |
+==========+=====================================+============================================+
| streams | streaming=True[,local=[True|False]] | type='streaming'[,distributed=[true|false] |
+----------+-------------------------------------+--------------------------------------------+
| events | retainsevents=True, streaming=False | type='eventing' |
+----------+-------------------------------------+--------------------------------------------+
| reports | streaming=False | type='reporting' |
+----------+-------------------------------------+--------------------------------------------+
Only streaming commands may be distributed to indexers. By default generating commands are configured to run
locally in the streams pipeline and will run under either SCP 1 or SCP 2.
.. code-block:: python
@Configuration()
class StreamingGeneratingCommand(GeneratingCommand)
...
How you configure your command to run on a different pipeline or in a distributed fashion depends on what SCP
protocol versions you wish to support. You must be sure to configure your command consistently for each protocol,
if you wish to support both protocol versions correctly.
.. _chart: http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Chart
.. _cluster: http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Cluster
.. _contingency: http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Contingency
.. _dedup: http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Dedup
.. _rex: http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Rex
.. _sort: http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Sort
.. _stats: http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Stats
Distributed Generating command
==============================
Commands configured like this will run as the first command on search heads and/or indexers on the streams pipeline.
+----------+---------------------------------------------------+---------------------------------------------------+
| Pipeline | SCP 1 | SCP 2 |
+==========+===================================================+===================================================+
| streams | 1. Add this line to your command's stanza in | 1. Add this configuration setting to your code: |
| | | |
| | default/commands.conf. | .. code-block:: python |
| | .. code-block:: python | @Configuration(distributed=True) |
| | local = false | class SomeCommand(GeneratingCommand) |
| | | ... |
| | 2. Restart splunk | |
| | | 2. You are good to go; no need to restart Splunk |
+----------+---------------------------------------------------+---------------------------------------------------+
Eventing Generating command
===========================
Generating commands configured like this will run as the first command on a search head on the events pipeline.
+----------+---------------------------------------------------+---------------------------------------------------+
| Pipeline | SCP 1 | SCP 2 |
+==========+===================================================+===================================================+
| events | You have a choice. Add these configuration | Add this configuration setting to your command |
| | settings to your command class: | setting to your command class: |
| | | |
| | .. code-block:: python | .. code-block:: python |
| | @Configuration( | @Configuration(type='eventing') |
| | retainsevents=True, streaming=False) | class SomeCommand(GeneratingCommand) |
| | class SomeCommand(GeneratingCommand) | ... |
| | ... | |
| | | |
| | Or add these lines to default/commands.conf: | |
| | | |
| | .. code-block:: | |
| | retains events = true | |
| | streaming = false | |
+----------+---------------------------------------------------+---------------------------------------------------+
Configure your command class like this, if you wish to support both protocols:
.. code-block:: python
@Configuration(type='eventing', retainsevents=True, streaming=False)
class SomeCommand(GeneratingCommand)
...
You might also consider adding these lines to commands.conf instead of adding them to your command class:
.. code-block:: python
retains events = false
streaming = false
Reporting Generating command
============================
Commands configured like this will run as the first command on a search head on the reports pipeline.
+----------+---------------------------------------------------+---------------------------------------------------+
| Pipeline | SCP 1 | SCP 2 |
+==========+===================================================+===================================================+
| events | You have a choice. Add these configuration | Add this configuration setting to your command |
| | settings to your command class: | setting to your command class: |
| | | |
| | .. code-block:: python | .. code-block:: python |
| | @Configuration(retainsevents=False) | @Configuration(type='reporting') |
| | class SomeCommand(GeneratingCommand) | class SomeCommand(GeneratingCommand) |
| | ... | ... |
| | | |
| | Or add this lines to default/commands.conf: | |
| | | |
| | .. code-block:: | |
| | retains events = false | |
| | streaming = false | |
+----------+---------------------------------------------------+---------------------------------------------------+
Configure your command class like this, if you wish to support both protocols:
.. code-block:: python
@Configuration(type='reporting', streaming=False)
class SomeCommand(GeneratingCommand)
...
You might also consider adding these lines to commands.conf instead of adding them to your command class:
.. code-block:: python
retains events = false
streaming = false
"""
# region Methods
def generate(self):
""" A generator that yields records to the Splunk processing pipeline
You must override this method.
"""
raise NotImplementedError('GeneratingCommand.generate(self)')
def _execute(self, ifile, process):
""" Execution loop
:param ifile: Input file object. Unused.
:type ifile: file
:return: `None`.
"""
self._record_writer.write_records(self.generate())
self.finish()
# endregion
# region Types
class ConfigurationSettings(SearchCommand.ConfigurationSettings):
""" Represents the configuration settings for a :code:`GeneratingCommand` class.
"""
# region SCP v1/v2 Properties
generating = ConfigurationSetting(readonly=True, value=True, doc='''
Tells Splunk that this command generates events, but does not process inputs.
Generating commands must appear at the front of the search pipeline identified by :meth:`type`.
Fixed: :const:`True`
Supported by: SCP 1, SCP 2
''')
# endregion
# region SCP v1 Properties
generates_timeorder = ConfigurationSetting(doc='''
:const:`True`, if the command generates new events.
Default: :const:`False`
Supported by: SCP 1
''')
local = ConfigurationSetting(doc='''
:const:`True`, if the command should run locally on the search head.
Default: :const:`False`
Supported by: SCP 1
''')
retainsevents = ConfigurationSetting(doc='''
:const:`True`, if the command retains events the way the sort, dedup, and cluster commands do, or whether it
transforms them the way the stats command does.
Default: :const:`False`
Supported by: SCP 1
''')
streaming = ConfigurationSetting(doc='''
:const:`True`, if the command is streamable.
Default: :const:`True`
Supported by: SCP 1
''')
# endregion
# region SCP v2 Properties
distributed = ConfigurationSetting(value=False, doc='''
True, if this command should be distributed to indexers.
This value is ignored unless :meth:`type` is equal to :const:`streaming`. It is only this command type that
may be distributed.
Default: :const:`False`
Supported by: SCP 2
''')
type = ConfigurationSetting(value='streaming', doc='''
A command type name.
==================== ======================================================================================
Value Description
-------------------- --------------------------------------------------------------------------------------
:const:`'eventing'` Runs as the first command in the Splunk events pipeline. Cannot be distributed.
:const:`'reporting'` Runs as the first command in the Splunk reports pipeline. Cannot be distributed.
:const:`'streaming'` Runs as the first command in the Splunk streams pipeline. May be distributed.
==================== ======================================================================================
Default: :const:`'streaming'`
Supported by: SCP 2
''')
# endregion
# region Methods
@classmethod
def fix_up(cls, command):
""" Verifies :code:`command` class structure.
"""
if command.generate == GeneratingCommand.generate:
raise AttributeError('No GeneratingCommand.generate override')
def iteritems(self):
iteritems = SearchCommand.ConfigurationSettings.iteritems(self)
version = self.command.protocol_version
if version == 2:
iteritems = ifilter(lambda (name, value): name != 'distributed', iteritems)
if self.distributed and self.type == 'streaming':
iteritems = imap(
lambda (name, value): (name, 'stateful') if name == 'type' else (name, value), iteritems)
return iteritems
pass
# endregion
pass
# endregion
package/bin/splunklib/searchcommands/internals.py
Executable
+786
View File
@@ -0,0 +1,786 @@
# coding=utf-8
#
# Copyright © 2011-2015 Splunk, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"): you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from __future__ import absolute_import, division, print_function, unicode_literals
from collections import deque, namedtuple
try:
from collections import OrderedDict # must be python 2.7
except ImportError:
from ..ordereddict import OrderedDict
from cStringIO import StringIO
from itertools import chain, imap
from json import JSONDecoder, JSONEncoder
from json.encoder import encode_basestring_ascii as json_encode_string
from urllib import unquote
import csv
import gzip
import os
import re
import sys
from . import environment
csv.field_size_limit(10485760) # The default value is 128KB; upping to 10MB. See SPL-12117 for background on this issue
if sys.platform == 'win32':
# Work around the fact that on Windows '\n' is mapped to '\r\n'. The typical solution is to simply open files in
# binary mode, but stdout is already open, thus this hack. 'CPython' and 'PyPy' work differently. We assume that
# all other Python implementations are compatible with 'CPython'. This might or might not be a valid assumption.
from platform import python_implementation
implementation = python_implementation()
fileno = sys.stdout.fileno()
if implementation == 'PyPy':
sys.stdout = os.fdopen(fileno, 'wb', 0)
else:
from msvcrt import setmode
setmode(fileno, os.O_BINARY)
class CommandLineParser(object):
""" Parses the arguments to a search command.
A search command line is described by the following syntax.
**Syntax**::
command = command-name *[wsp option] *[wsp [dquote] field-name [dquote]]
command-name = alpha *( alpha / digit )
option = option-name [wsp] "=" [wsp] option-value
option-name = alpha *( alpha / digit / "_" )
option-value = word / quoted-string
word = 1*( %01-%08 / %0B / %0C / %0E-1F / %21 / %23-%FF ) ; Any character but DQUOTE and WSP
quoted-string = dquote *( word / wsp / "\" dquote / dquote dquote ) dquote
field-name = ( "_" / alpha ) *( alpha / digit / "_" / "." / "-" )
**Note:**
This syntax is constrained to an 8-bit character set.
**Note:**
This syntax does not show that `field-name` values may be comma-separated when in fact they can be. This is
because Splunk strips commas from the command line. A custom search command will never see them.
**Example:**
countmatches fieldname = word_count pattern = \w+ some_text_field
Option names are mapped to properties in the targeted ``SearchCommand``. It is the responsibility of the property
setters to validate the values they receive. Property setters may also produce side effects. For example,
setting the built-in `log_level` immediately changes the `log_level`.
"""
@classmethod
def parse(cls, command, argv):
""" Splits an argument list into an options dictionary and a fieldname
list.
The argument list, `argv`, must be of the form::
*[option]... *[<field-name>]
Options are validated and assigned to items in `command.options`. Field names are validated and stored in the
list of `command.fieldnames`.
#Arguments:
:param command: Search command instance.
:type command: ``SearchCommand``
:param argv: List of search command arguments.
:type argv: ``list``
:return: ``None``
#Exceptions:
``SyntaxError``: Argument list is incorrectly formed.
``ValueError``: Unrecognized option/field name, or an illegal field value.
"""
debug = environment.splunklib_logger.debug
command_class = type(command).__name__
# Prepare
debug('Parsing %s command line: %r', command_class, argv)
command.fieldnames = None
command.options.reset()
argv = ' '.join(argv)
command_args = cls._arguments_re.match(argv)
if command_args is None:
raise SyntaxError('Syntax error: {}'.format(argv))
# Parse options
for option in cls._options_re.finditer(command_args.group('options')):
name, value = option.group('name'), option.group('value')
if name not in command.options:
raise ValueError(
'Unrecognized {} command option: {}={}'.format(command.name, name, json_encode_string(value)))
command.options[name].value = cls.unquote(value)
missing = command.options.get_missing()
if missing is not None:
if len(missing) > 1:
raise ValueError(
'Values for these {} command options are required: {}'.format(command.name, ', '.join(missing)))
raise ValueError('A value for {} command option {} is required'.format(command.name, missing[0]))
# Parse field names
fieldnames = command_args.group('fieldnames')
if fieldnames is None:
command.fieldnames = []
else:
command.fieldnames = [cls.unquote(value.group(0)) for value in cls._fieldnames_re.finditer(fieldnames)]
debug(' %s: %s', command_class, command)
@classmethod
def unquote(cls, string):
""" Removes quotes from a quoted string.
Splunk search command quote rules are applied. The enclosing double-quotes, if present, are removed. Escaped
double-quotes ('\"' or '""') are replaced by a single double-quote ('"').
**NOTE**
We are not using a json.JSONDecoder because Splunk quote rules are different than JSON quote rules. A
json.JSONDecoder does not recognize a pair of double-quotes ('""') as an escaped quote ('"') and will
decode single-quoted strings ("'") in addition to double-quoted ('"') strings.
"""
if len(string) == 0:
return ''
if string[0] == '"':
if len(string) == 1 or string[-1] != '"':
raise SyntaxError('Poorly formed string literal: ' + string)
string = string[1:-1]
if len(string) == 0:
return ''
def replace(match):
value = match.group(0)
if value == '""':
return '"'
if len(value) < 2:
raise SyntaxError('Poorly formed string literal: ' + string)
return value[1]
result = re.sub(cls._escaped_character_re, replace, string)
return result
# region Class variables
_arguments_re = re.compile(r"""
^\s*
(?P<options> # Match a leading set of name/value pairs
(?:
(?:(?=\w)[^\d]\w*) # name
\s*=\s* # =
(?:"(?:\\.|""|[^"])*"|(?:\\.|[^\s"])+)\s* # value
)*
)\s*
(?P<fieldnames> # Match a trailing set of field names
(?:
(?:"(?:\\.|""|[^"])*"|(?:\\.|[^\s"])+)\s*
)*
)\s*$
""", re.VERBOSE | re.UNICODE)
_escaped_character_re = re.compile(r'(\\.|""|[\\"])')
_fieldnames_re = re.compile(r"""("(?:\\.|""|[^"])+"|(?:\\.|[^\s"])+)""")
_options_re = re.compile(r"""
# Captures a set of name/value pairs when used with re.finditer
(?P<name>(?:(?=\w)[^\d]\w*)) # name
\s*=\s* # =
(?P<value>"(?:\\.|""|[^"])*"|(?:\\.|[^\s"])+) # value
""", re.VERBOSE | re.UNICODE)
# endregion
class ConfigurationSettingsType(type):
""" Metaclass for constructing ConfigurationSettings classes.
Instances of :class:`ConfigurationSettingsType` construct :class:`ConfigurationSettings` classes from classes from
a base :class:`ConfigurationSettings` class and a dictionary of configuration settings. The settings in the
dictionary are validated against the settings in the base class. You cannot add settings, you can only change their
backing-field values and you cannot modify settings without backing-field values. These are considered fixed
configuration setting values.
This is an internal class used in two places:
+ :meth:`decorators.Configuration.__call__`
Adds a ConfigurationSettings attribute to a :class:`SearchCommand` class.
+ :meth:`reporting_command.ReportingCommand.fix_up`
Adds a ConfigurationSettings attribute to a :meth:`ReportingCommand.map` method, if there is one.
"""
def __new__(mcs, module, name, bases):
mcs = super(ConfigurationSettingsType, mcs).__new__(mcs, name, bases, {})
return mcs
def __init__(cls, module, name, bases):
super(ConfigurationSettingsType, cls).__init__(name, bases, None)
cls.__module__ = module
@staticmethod
def validate_configuration_setting(specification, name, value):
if not isinstance(value, specification.type):
if isinstance(specification.type, type):
type_names = specification.type.__name__
else:
type_names = ', '.join(imap(lambda t: t.__name__, specification.type))
raise ValueError('Expected {} value, not {}={}'.format(type_names, name, repr(value)))
if specification.constraint and not specification.constraint(value):
raise ValueError('Illegal value: {}={}'.format(name, repr(value)))
return value
specification = namedtuple(
b'ConfigurationSettingSpecification', (
b'type',
b'constraint',
b'supporting_protocols'))
# P1 [ ] TODO: Review ConfigurationSettingsType.specification_matrix for completeness and correctness
specification_matrix = {
'clear_required_fields': specification(
type=bool,
constraint=None,
supporting_protocols=[1]),
'distributed': specification(
type=bool,
constraint=None,
supporting_protocols=[2]),
'generates_timeorder': specification(
type=bool,
constraint=None,
supporting_protocols=[1]),
'generating': specification(
type=bool,
constraint=None,
supporting_protocols=[1, 2]),
'local': specification(
type=bool,
constraint=None,
supporting_protocols=[1]),
'maxinputs': specification(
type=int,
constraint=lambda value: 0 <= value <= sys.maxint,
supporting_protocols=[2]),
'overrides_timeorder': specification(
type=bool,
constraint=None,
supporting_protocols=[1]),
'required_fields': specification(
type=(list, set, tuple),
constraint=None,
supporting_protocols=[1, 2]),
'requires_preop': specification(
type=bool,
constraint=None,
supporting_protocols=[1]),
'retainsevents': specification(
type=bool,
constraint=None,
supporting_protocols=[1]),
'run_in_preview': specification(
type=bool,
constraint=None,
supporting_protocols=[2]),
'streaming': specification(
type=bool,
constraint=None,
supporting_protocols=[1]),
'streaming_preop': specification(
type=(bytes, unicode),
constraint=None,
supporting_protocols=[1, 2]),
'type': specification(
type=(bytes, unicode),
constraint=lambda value: value in ('eventing', 'reporting', 'streaming'),
supporting_protocols=[2])}
class CsvDialect(csv.Dialect):
""" Describes the properties of Splunk CSV streams """
delimiter = b','
quotechar = b'"'
doublequote = True
skipinitialspace = False
lineterminator = b'\r\n'
quoting = csv.QUOTE_MINIMAL
class InputHeader(dict):
""" Represents a Splunk input header as a collection of name/value pairs.
"""
def __str__(self):
return '\n'.join([name + ':' + value for name, value in self.iteritems()])
def read(self, ifile):
""" Reads an input header from an input file.
The input header is read as a sequence of *<name>***:***<value>* pairs separated by a newline. The end of the
input header is signalled by an empty line or an end-of-file.
:param ifile: File-like object that supports iteration over lines.
"""
name, value = None, None
for line in ifile:
if line == '\n':
break
item = line.split(':', 1)
if len(item) == 2:
# start of a new item
if name is not None:
self[name] = value[:-1] # value sans trailing newline
name, value = item[0], unquote(item[1])
elif name is not None:
# continuation of the current item
value += unquote(line)
if name is not None: self[name] = value[:-1] if value[-1] == '\n' else value
Message = namedtuple(b'Message', (b'type', b'text'))
class MetadataDecoder(JSONDecoder):
def __init__(self):
JSONDecoder.__init__(self, object_hook=self._object_hook)
@staticmethod
def _object_hook(dictionary):
object_view = ObjectView(dictionary)
stack = deque()
stack.append((None, None, dictionary))
while len(stack):
instance, member_name, dictionary = stack.popleft()
for name, value in dictionary.iteritems():
if isinstance(value, dict):
stack.append((dictionary, name, value))
if instance is not None:
instance[member_name] = ObjectView(dictionary)
return object_view
class MetadataEncoder(JSONEncoder):
def __init__(self):
JSONEncoder.__init__(self, separators=MetadataEncoder._separators)
def default(self, o):
return o.__dict__ if isinstance(o, ObjectView) else JSONEncoder.default(self, o)
_separators = (',', ':')
class ObjectView(object):
def __init__(self, dictionary):
self.__dict__ = dictionary
def __repr__(self):
return repr(self.__dict__)
def __str__(self):
return str(self.__dict__)
class Recorder(object):
def __init__(self, path, f):
self._recording = gzip.open(path + '.gz', 'wb')
self._file = f
def __getattr__(self, name):
return getattr(self._file, name)
def __iter__(self):
for line in self._file:
self._recording.write(line)
self._recording.flush()
yield line
def read(self, size=None):
value = self._file.read() if size is None else self._file.read(size)
self._recording.write(value)
self._recording.flush()
return value
def readline(self, size=None):
value = self._file.readline() if size is None else self._file.readline(size)
if len(value) > 0:
self._recording.write(value)
self._recording.flush()
return value
def record(self, *args):
for arg in args:
self._recording.write(arg)
def write(self, text):
self._recording.write(text)
self._file.write(text)
self._recording.flush()
class RecordWriter(object):
def __init__(self, ofile, maxresultrows=None):
self._maxresultrows = 50000 if maxresultrows is None else maxresultrows
self._ofile = ofile
self._fieldnames = None
self._buffer = StringIO()
self._writer = csv.writer(self._buffer, dialect=CsvDialect)
self._writerow = self._writer.writerow
self._finished = False
self._flushed = False
self._inspector = OrderedDict()
self._chunk_count = 0
self._record_count = 0
self._total_record_count = 0L
@property
def is_flushed(self):
return self._flushed
@is_flushed.setter
def is_flushed(self, value):
self._flushed = True if value else False
@property
def ofile(self):
return self._ofile
@ofile.setter
def ofile(self, value):
self._ofile = value
def flush(self, finished=None, partial=None):
assert finished is None or isinstance(finished, bool)
assert partial is None or isinstance(partial, bool)
assert not (finished is None and partial is None)
assert finished is None or partial is None
self._ensure_validity()
def write_message(self, message_type, message_text, *args, **kwargs):
self._ensure_validity()
self._inspector.setdefault('messages', []).append((message_type, message_text.format(*args, **kwargs)))
def write_record(self, record):
self._ensure_validity()
self._write_record(record)
def write_records(self, records):
self._ensure_validity()
write_record = self._write_record
for record in records:
write_record(record)
def _clear(self):
self._buffer.reset()
self._buffer.truncate()
self._inspector.clear()
self._record_count = 0
self._flushed = False
def _ensure_validity(self):
if self._finished is True:
assert self._record_count == 0 and len(self._inspector) == 0
raise RuntimeError('I/O operation on closed record writer')
def _write_record(self, record):
fieldnames = self._fieldnames
if fieldnames is None:
self._fieldnames = fieldnames = record.keys()
value_list = imap(lambda fn: unicode(fn).encode('utf-8'), fieldnames)
value_list = imap(lambda fn: (fn, b'__mv_' + fn), value_list)
self._writerow(list(chain.from_iterable(value_list)))
get_value = record.get
values = []
for fieldname in fieldnames:
value = get_value(fieldname, None)
if value is None:
values += (None, None)
continue
value_t = type(value)
if issubclass(value_t, (list, tuple)):
if len(value) == 0:
values += (None, None)
continue
if len(value) > 1:
value_list = value
sv = b''
mv = b'$'
for value in value_list:
if value is None:
sv += b'\n'
mv += b'$;$'
continue
value_t = type(value)
if value_t is not bytes:
if value_t is bool:
value = str(value.real)
elif value_t is unicode:
value = value.encode('utf-8', errors='backslashreplace')
elif value_t is int or value_t is long or value_t is float or value_t is complex:
value = str(value)
elif issubclass(value_t, (dict, list, tuple)):
value = str(''.join(RecordWriter._iterencode_json(value, 0)))
else:
value = repr(value).encode('utf-8', errors='backslashreplace')
sv += value + b'\n'
mv += value.replace(b'$', b'$$') + b'$;$'
values += (sv[:-1], mv[:-2])
continue
value = value[0]
value_t = type(value)
if value_t is bool:
values += (str(value.real), None)
continue
if value_t is bytes:
values += (value, None)
continue
if value_t is unicode:
values += (value.encode('utf-8', errors='backslashreplace'), None)
continue
if value_t is int or value_t is long or value_t is float or value_t is complex:
values += (str(value), None)
continue
if issubclass(value_t, dict):
values += (str(''.join(RecordWriter._iterencode_json(value, 0))), None)
continue
values += (repr(value).encode('utf-8', errors='backslashreplace'), None)
self._writerow(values)
self._record_count += 1
if self._record_count >= self._maxresultrows:
self.flush(partial=True)
try:
# noinspection PyUnresolvedReferences
from _json import make_encoder
except ImportError:
# We may be running under PyPy 2.5 which does not include the _json module
_iterencode_json = JSONEncoder(separators=(',', ':')).iterencode
else:
# Creating _iterencode_json this way yields a two-fold performance improvement on Python 2.7.9 and 2.7.10
from json.encoder import encode_basestring_ascii
@staticmethod
def _default(o):
raise TypeError(repr(o) + ' is not JSON serializable')
_iterencode_json = make_encoder(
{}, # markers (for detecting circular references)
_default, # object_encoder
encode_basestring_ascii, # string_encoder
None, # indent
':', ',', # separators
False, # sort_keys
False, # skip_keys
True # allow_nan
)
del make_encoder
class RecordWriterV1(RecordWriter):
def flush(self, finished=None, partial=None):
RecordWriter.flush(self, finished, partial) # validates arguments and the state of this instance
if self._record_count > 0 or (self._chunk_count == 0 and 'messages' in self._inspector):
messages = self._inspector.get('messages')
write = self._ofile.write
if self._chunk_count == 0:
# Messages are written to the messages header when we write the first chunk of data
# Guarantee: These messages are displayed by splunkweb and the job inspector
if messages is not None:
message_level = RecordWriterV1._message_level.get
for level, text in messages:
write(message_level(level, level))
write('=')
write(text)
write('\r\n')
write('\r\n')
elif messages is not None:
# Messages are written to the messages header when we write subsequent chunks of data
# Guarantee: These messages are displayed by splunkweb and the job inspector, if and only if the
# command is configured with
#
# stderr_dest = message
#
# stderr_dest is a static configuration setting. This means that it can only be set in commands.conf.
# It cannot be set in code.
stderr = sys.stderr
for level, text in messages:
print(level, text, file=stderr)
write(self._buffer.getvalue())
self._clear()
self._chunk_count += 1
self._total_record_count += self._record_count
self._finished = finished is True
_message_level = {
'DEBUG': 'debug_message',
'ERROR': 'error_message',
'FATAL': 'error_message',
'INFO': 'info_message',
'WARN': 'warn_message'
}
class RecordWriterV2(RecordWriter):
def flush(self, finished=None, partial=None):
RecordWriter.flush(self, finished, partial) # validates arguments and the state of this instance
inspector = self._inspector
if self._flushed is False:
self._total_record_count += self._record_count
self._chunk_count += 1
# TODO: DVPL-6448: splunklib.searchcommands | Add support for partial: true when it is implemented in
# ChunkedExternProcessor (See SPL-103525)
#
# We will need to replace the following block of code with this block:
#
# metadata = [
# ('inspector', self._inspector if len(self._inspector) else None),
# ('finished', finished),
# ('partial', partial)]
if len(inspector) == 0:
inspector = None
if partial is True:
finished = False
metadata = [item for item in ('inspector', inspector), ('finished', finished)]
self._write_chunk(metadata, self._buffer.getvalue())
self._clear()
elif finished is True:
self._write_chunk((('finished', True),), '')
self._finished = finished is True
def write_metadata(self, configuration):
self._ensure_validity()
metadata = chain(configuration.iteritems(), (('inspector', self._inspector if self._inspector else None),))
self._write_chunk(metadata, '')
self._ofile.write('\n')
self._clear()
def write_metric(self, name, value):
self._ensure_validity()
self._inspector['metric.' + name] = value
def _clear(self):
RecordWriter._clear(self)
self._fieldnames = None
def _write_chunk(self, metadata, body):
if metadata:
metadata = str(''.join(self._iterencode_json(dict([(n, v) for n, v in metadata if v is not None]), 0)))
metadata_length = len(metadata)
else:
metadata_length = 0
body_length = len(body)
if not (metadata_length > 0 or body_length > 0):
return
start_line = b'chunked 1.0,' + bytes(metadata_length) + b',' + bytes(body_length) + b'\n'
write = self._ofile.write
write(start_line)
write(metadata)
write(body)
self._ofile.flush()
self._flushed = False
package/bin/splunklib/searchcommands/reporting_command.py
Executable
+280
View File
@@ -0,0 +1,280 @@
# coding=utf-8
#
# Copyright © 2011-2015 Splunk, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"): you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from __future__ import absolute_import, division, print_function, unicode_literals
from itertools import chain
from .internals import ConfigurationSettingsType, json_encode_string
from .decorators import ConfigurationSetting, Option
from .streaming_command import StreamingCommand
from .search_command import SearchCommand
from .validators import Set
class ReportingCommand(SearchCommand):
""" Processes search result records and generates a reporting data structure.
Reporting search commands run as either reduce or map/reduce operations. The reduce part runs on a search head and
is responsible for processing a single chunk of search results to produce the command's reporting data structure.
The map part is called a streaming preop. It feeds the reduce part with partial results and by default runs on the
search head and/or one or more indexers.
You must implement a :meth:`reduce` method as a generator function that iterates over a set of event records and
yields a reporting data structure. You may implement a :meth:`map` method as a generator function that iterates
over a set of event records and yields :class:`dict` or :class:`list(dict)` instances.
ReportingCommand configuration
==============================
Configure the :meth:`map` operation using a Configuration decorator on your :meth:`map` method. Configure it like
you would a :class:`StreamingCommand`. Configure the :meth:`reduce` operation using a Configuration decorator on
your :meth:`ReportingCommand` class.
You can configure your command for operation under Search Command Protocol (SCP) version 1 or 2. SCP 2 requires
Splunk 6.3 or later.
"""
# region Special methods
def __init__(self):
SearchCommand.__init__(self)
# endregion
# region Options
phase = Option(doc='''
**Syntax:** phase=[map|reduce]
**Description:** Identifies the phase of the current map-reduce operation.
''', default='reduce', validate=Set('map', 'reduce'))
# endregion
# region Methods
def map(self, records):
""" Override this method to compute partial results.
:param records:
:type records:
You must override this method, if :code:`requires_preop=True`.
"""
return NotImplemented
def prepare(self):
phase = self.phase
if phase == 'map':
# noinspection PyUnresolvedReferences
self._configuration = self.map.ConfigurationSettings(self)
return
if phase == 'reduce':
streaming_preop = chain((self.name, 'phase="map"', str(self._options)), self.fieldnames)
self._configuration.streaming_preop = ' '.join(streaming_preop)
return
raise RuntimeError('Unrecognized reporting command phase: {}'.format(json_encode_string(unicode(phase))))
def reduce(self, records):
""" Override this method to produce a reporting data structure.
You must override this method.
"""
raise NotImplementedError('reduce(self, records)')
def _execute(self, ifile, process):
SearchCommand._execute(self, ifile, getattr(self, self.phase))
# endregion
# region Types
class ConfigurationSettings(SearchCommand.ConfigurationSettings):
""" Represents the configuration settings for a :code:`ReportingCommand`.
"""
# region SCP v1/v2 Properties
required_fields = ConfigurationSetting(doc='''
List of required fields for this search which back-propagates to the generating search.
Setting this value enables selected fields mode under SCP 2. Under SCP 1 you must also specify
:code:`clear_required_fields=True` to enable selected fields mode. To explicitly select all fields,
specify a value of :const:`['*']`. No error is generated if a specified field is missing.
Default: :const:`None`, which implicitly selects all fields.
Supported by: SCP 1, SCP 2
''')
requires_preop = ConfigurationSetting(doc='''
Indicates whether :meth:`ReportingCommand.map` is required for proper command execution.
If :const:`True`, :meth:`ReportingCommand.map` is guaranteed to be called. If :const:`False`, Splunk
considers it to be an optimization that may be skipped.
Default: :const:`False`
Supported by: SCP 1, SCP 2
''')
streaming_preop = ConfigurationSetting(doc='''
Denotes the requested streaming preop search string.
Computed.
Supported by: SCP 1, SCP 2
''')
# endregion
# region SCP v1 Properties
clear_required_fields = ConfigurationSetting(doc='''
:const:`True`, if required_fields represent the *only* fields required.
If :const:`False`, required_fields are additive to any fields that may be required by subsequent commands.
In most cases, :const:`True` is appropriate for reporting commands.
Default: :const:`True`
Supported by: SCP 1
''')
retainsevents = ConfigurationSetting(readonly=True, value=False, doc='''
Signals that :meth:`ReportingCommand.reduce` transforms _raw events to produce a reporting data structure.
Fixed: :const:`False`
Supported by: SCP 1
''')
streaming = ConfigurationSetting(readonly=True, value=False, doc='''
Signals that :meth:`ReportingCommand.reduce` runs on the search head.
Fixed: :const:`False`
Supported by: SCP 1
''')
# endregion
# region SCP v2 Properties
maxinputs = ConfigurationSetting(doc='''
Specifies the maximum number of events that can be passed to the command for each invocation.
This limit cannot exceed the value of `maxresultrows` in limits.conf_. Under SCP 1 you must specify this
value in commands.conf_.
Default: The value of `maxresultrows`.
Supported by: SCP 2
.. _limits.conf: http://docs.splunk.com/Documentation/Splunk/latest/admin/Limitsconf
''')
run_in_preview = ConfigurationSetting(doc='''
:const:`True`, if this command should be run to generate results for preview; not wait for final output.
This may be important for commands that have side effects (e.g., outputlookup).
Default: :const:`True`
Supported by: SCP 2
''')
type = ConfigurationSetting(readonly=True, value='reporting', doc='''
Command type name.
Fixed: :const:`'reporting'`.
Supported by: SCP 2
''')
# endregion
# region Methods
@classmethod
def fix_up(cls, command):
""" Verifies :code:`command` class structure and configures the :code:`command.map` method.
Verifies that :code:`command` derives from :class:`ReportingCommand` and overrides
:code:`ReportingCommand.reduce`. It then configures :code:`command.reduce`, if an overriding implementation
of :code:`ReportingCommand.reduce` has been provided.
:param command: :code:`ReportingCommand` class
Exceptions:
:code:`TypeError` :code:`command` class is not derived from :code:`ReportingCommand`
:code:`AttributeError` No :code:`ReportingCommand.reduce` override
"""
if not issubclass(command, ReportingCommand):
raise TypeError('{} is not a ReportingCommand'.format( command))
if command.reduce == ReportingCommand.reduce:
raise AttributeError('No ReportingCommand.reduce override')
if command.map == ReportingCommand.map:
cls._requires_preop = False
return
f = vars(command)[b'map'] # Function backing the map method
# EXPLANATION OF PREVIOUS STATEMENT: There is no way to add custom attributes to methods. See [Why does
# setattr fail on a method](http://stackoverflow.com/questions/7891277/why-does-setattr-fail-on-a-bound-method) for a discussion of this issue.
try:
settings = f._settings
except AttributeError:
f.ConfigurationSettings = StreamingCommand.ConfigurationSettings
return
# Create new StreamingCommand.ConfigurationSettings class
module = command.__module__ + b'.' + command.__name__ + b'.map'
name = b'ConfigurationSettings'
bases = (StreamingCommand.ConfigurationSettings,)
f.ConfigurationSettings = ConfigurationSettingsType(module, name, bases)
ConfigurationSetting.fix_up(f.ConfigurationSettings, settings)
del f._settings
pass
# endregion
pass
# endregion
package/bin/splunklib/searchcommands/search_command.py
Executable
+1101
View File
File diff suppressed because it is too large Load Diff
package/bin/splunklib/searchcommands/streaming_command.py
Executable
+188
View File
@@ -0,0 +1,188 @@
# coding=utf-8
#
# Copyright 2011-2015 Splunk, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"): you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from __future__ import absolute_import, division, print_function, unicode_literals
from itertools import ifilter, imap
from .decorators import ConfigurationSetting
from .search_command import SearchCommand
class StreamingCommand(SearchCommand):
""" Applies a transformation to search results as they travel through the streams pipeline.
Streaming commands typically filter, augment, or update, search result records. Splunk will send them in batches of
up to 50,000 records. Hence, a search command must be prepared to be invoked many times during the course of
pipeline processing. Each invocation should produce a set of results independently usable by downstream processors.
By default Splunk may choose to run a streaming command locally on a search head and/or remotely on one or more
indexers concurrently. The size and frequency of the search result batches sent to the command will vary based
on scheduling considerations.
StreamingCommand configuration
==============================
You can configure your command for operation under Search Command Protocol (SCP) version 1 or 2. SCP 2 requires
Splunk 6.3 or later.
"""
# region Methods
def stream(self, records):
""" Generator function that processes and yields event records to the Splunk stream pipeline.
You must override this method.
"""
raise NotImplementedError('StreamingCommand.stream(self, records)')
def _execute(self, ifile, process):
SearchCommand._execute(self, ifile, self.stream)
# endregion
class ConfigurationSettings(SearchCommand.ConfigurationSettings):
""" Represents the configuration settings that apply to a :class:`StreamingCommand`.
"""
# region SCP v1/v2 properties
required_fields = ConfigurationSetting(doc='''
List of required fields for this search which back-propagates to the generating search.
Setting this value enables selected fields mode under SCP 2. Under SCP 1 you must also specify
:code:`clear_required_fields=True` to enable selected fields mode. To explicitly select all fields,
specify a value of :const:`['*']`. No error is generated if a specified field is missing.
Default: :const:`None`, which implicitly selects all fields.
Supported by: SCP 1, SCP 2
''')
# endregion
# region SCP v1 properties
clear_required_fields = ConfigurationSetting(doc='''
:const:`True`, if required_fields represent the *only* fields required.
If :const:`False`, required_fields are additive to any fields that may be required by subsequent commands.
In most cases, :const:`False` is appropriate for streaming commands.
Default: :const:`False`
Supported by: SCP 1
''')
local = ConfigurationSetting(doc='''
:const:`True`, if the command should run locally on the search head.
Default: :const:`False`
Supported by: SCP 1
''')
overrides_timeorder = ConfigurationSetting(doc='''
:const:`True`, if the command changes the order of events with respect to time.
Default: :const:`False`
Supported by: SCP 1
''')
streaming = ConfigurationSetting(readonly=True, value=True, doc='''
Specifies that the command is streamable.
Fixed: :const:`True`
Supported by: SCP 1
''')
# endregion
# region SCP v2 Properties
distributed = ConfigurationSetting(value=True, doc='''
:const:`True`, if this command should be distributed to indexers.
Under SCP 1 you must either specify `local = False` or include this line in commands.conf_, if this command
should be distributed to indexers.
..code:
local = true
Default: :const:`True`
Supported by: SCP 2
.. commands.conf_: http://docs.splunk.com/Documentation/Splunk/latest/Admin/Commandsconf
''')
maxinputs = ConfigurationSetting(doc='''
Specifies the maximum number of events that can be passed to the command for each invocation.
This limit cannot exceed the value of `maxresultrows` in limits.conf. Under SCP 1 you must specify this
value in commands.conf_.
Default: The value of `maxresultrows`.
Supported by: SCP 2
''')
type = ConfigurationSetting(readonly=True, value='streaming', doc='''
Command type name.
Fixed: :const:`'streaming'`
Supported by: SCP 2
''')
# endregion
# region Methods
@classmethod
def fix_up(cls, command):
""" Verifies :code:`command` class structure.
"""
if command.stream == StreamingCommand.stream:
raise AttributeError('No StreamingCommand.stream override')
return
def iteritems(self):
iteritems = SearchCommand.ConfigurationSettings.iteritems(self)
version = self.command.protocol_version
if version == 1:
if self.required_fields is None:
iteritems = ifilter(lambda (name, value): name != 'clear_required_fields', iteritems)
else:
iteritems = ifilter(lambda (name, value): name != 'distributed', iteritems)
if self.distributed:
iteritems = imap(
lambda (name, value): (name, 'stateful') if name == 'type' else (name, value), iteritems)
return iteritems
# endregion
package/bin/splunklib/searchcommands/validators.py
Executable
+384
View File
@@ -0,0 +1,384 @@
# coding=utf-8
#
# Copyright 2011-2015 Splunk, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"): you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from __future__ import absolute_import, division, print_function, unicode_literals
from json.encoder import encode_basestring_ascii as json_encode_string
from collections import namedtuple
from cStringIO import StringIO
from io import open
import csv
import os
import re
class Validator(object):
""" Base class for validators that check and format search command options.
You must inherit from this class and override :code:`Validator.__call__` and
:code:`Validator.format`. :code:`Validator.__call__` should convert the
value it receives as argument and then return it or raise a
:code:`ValueError`, if the value will not convert.
:code:`Validator.format` should return a human readable version of the value
it receives as argument the same way :code:`str` does.
"""
def __call__(self, value):
raise NotImplementedError()
def format(self, value):
raise NotImplementedError()
class Boolean(Validator):
""" Validates Boolean option values.
"""
truth_values = {
'1': True, '0': False,
't': True, 'f': False,
'true': True, 'false': False,
'y': True, 'n': False,
'yes': True, 'no': False
}
def __call__(self, value):
if not (value is None or isinstance(value, bool)):
value = unicode(value).lower()
if value not in Boolean.truth_values:
raise ValueError('Unrecognized truth value: {0}'.format(value))
value = Boolean.truth_values[value]
return value
def format(self, value):
return None if value is None else 't' if value else 'f'
class Code(Validator):
""" Validates code option values.
This validator compiles an option value into a Python code object that can be executed by :func:`exec` or evaluated
by :func:`eval`. The value returned is a :func:`namedtuple` with two members: object, the result of compilation, and
source, the original option value.
"""
def __init__(self, mode='eval'):
"""
:param mode: Specifies what kind of code must be compiled; it can be :const:`'exec'`, if source consists of a
sequence of statements, :const:`'eval'`, if it consists of a single expression, or :const:`'single'` if it
consists of a single interactive statement. In the latter case, expression statements that evaluate to
something other than :const:`None` will be printed.
:type mode: unicode or bytes
"""
self._mode = mode
def __call__(self, value):
if value is None:
return None
try:
return Code.object(compile(value, 'string', self._mode), unicode(value))
except (SyntaxError, TypeError) as error:
raise ValueError(error.message)
def format(self, value):
return None if value is None else value.source
object = namedtuple(b'Code', (b'object', 'source'))
class Fieldname(Validator):
""" Validates field name option values.
"""
pattern = re.compile(r'''[_.a-zA-Z-][_.a-zA-Z0-9-]*$''')
def __call__(self, value):
if value is not None:
value = unicode(value)
if Fieldname.pattern.match(value) is None:
raise ValueError('Illegal characters in fieldname: {}'.format(value))
return value
def format(self, value):
return value
class File(Validator):
""" Validates file option values.
"""
def __init__(self, mode='rt', buffering=None, directory=None):
self.mode = mode
self.buffering = buffering
self.directory = File._var_run_splunk if directory is None else directory
def __call__(self, value):
if value is None:
return value
path = unicode(value)
if not os.path.isabs(path):
path = os.path.join(self.directory, path)
try:
value = open(path, self.mode) if self.buffering is None else open(path, self.mode, self.buffering)
except IOError as error:
raise ValueError('Cannot open {0} with mode={1} and buffering={2}: {3}'.format(
value, self.mode, self.buffering, error))
return value
def format(self, value):
return None if value is None else value.name
_var_run_splunk = os.path.join(
os.environ['SPLUNK_HOME'] if 'SPLUNK_HOME' in os.environ else os.getcwdu(), 'var', 'run', 'splunk')
class Integer(Validator):
""" Validates integer option values.
"""
def __init__(self, minimum=None, maximum=None):
if minimum is not None and maximum is not None:
def check_range(value):
if not (minimum <= value <= maximum):
raise ValueError('Expected integer in the range [{0},{1}], not {2}'.format(minimum, maximum, value))
return
elif minimum is not None:
def check_range(value):
if value < minimum:
raise ValueError('Expected integer in the range [{0},+∞], not {1}'.format(minimum, value))
return
elif maximum is not None:
def check_range(value):
if value > maximum:
raise ValueError('Expected integer in the range [-∞,{0}], not {1}'.format(maximum, value))
return
else:
def check_range(value):
return
self.check_range = check_range
return
def __call__(self, value):
if value is None:
return None
try:
value = long(value)
except ValueError:
raise ValueError('Expected integer value, not {}'.format(json_encode_string(value)))
self.check_range(value)
return value
def format(self, value):
return None if value is None else unicode(long(value))
class Duration(Validator):
""" Validates duration option values.
"""
def __call__(self, value):
if value is None:
return None
p = value.split(':', 2)
result = None
_60 = Duration._60
_unsigned = Duration._unsigned
try:
if len(p) == 1:
result = _unsigned(p[0])
if len(p) == 2:
result = 60 * _unsigned(p[0]) + _60(p[1])
if len(p) == 3:
result = 3600 * _unsigned(p[0]) + 60 * _60(p[1]) + _60(p[2])
except ValueError:
raise ValueError('Invalid duration value: {0}'.format(value))
return result
def format(self, value):
if value is None:
return None
value = int(value)
s = value % 60
m = value // 60 % 60
h = value // (60 * 60)
return '{0:02d}:{1:02d}:{2:02d}'.format(h, m, s)
_60 = Integer(0, 59)
_unsigned = Integer(0)
class List(Validator):
""" Validates a list of strings
"""
class Dialect(csv.Dialect):
""" Describes the properties of list option values. """
strict = True
delimiter = b','
quotechar = b'"'
doublequote = True
lineterminator = b'\n'
skipinitialspace = True
quoting = csv.QUOTE_MINIMAL
def __init__(self, validator=None):
if not (validator is None or isinstance(validator, Validator)):
raise ValueError('Expected a Validator instance or None for validator, not {}', repr(validator))
self._validator = validator
def __call__(self, value):
if value is None or isinstance(value, list):
return value
try:
value = csv.reader([value], self.Dialect).next()
except csv.Error as error:
raise ValueError(error)
if self._validator is None:
return value
try:
for index, item in enumerate(value):
value[index] = self._validator(item)
except ValueError as error:
raise ValueError('Could not convert item {}: {}'.format(index, error))
return value
def format(self, value):
output = StringIO()
writer = csv.writer(output, List.Dialect)
writer.writerow(value)
value = output.getvalue()
return value[:-1]
class Map(Validator):
""" Validates map option values.
"""
def __init__(self, **kwargs):
self.membership = kwargs
def __call__(self, value):
if value is None:
return None
value = unicode(value)
if value not in self.membership:
raise ValueError('Unrecognized value: {0}'.format(value))
return self.membership[value]
def format(self, value):
return None if value is None else self.membership.keys()[self.membership.values().index(value)]
class Match(Validator):
""" Validates that a value matches a regular expression pattern.
"""
def __init__(self, name, pattern, flags=0):
self.name = unicode(name)
self.pattern = re.compile(pattern, flags)
def __call__(self, value):
if value is None:
return None
value = unicode(value)
if self.pattern.match(value) is None:
raise ValueError('Expected {}, not {}'.format(self.name, json_encode_string(value)))
return value
def format(self, value):
return None if value is None else unicode(value)
class OptionName(Validator):
""" Validates option names.
"""
pattern = re.compile(r'''(?=\w)[^\d]\w*$''', re.UNICODE)
def __call__(self, value):
if value is not None:
value = unicode(value)
if OptionName.pattern.match(value) is None:
raise ValueError('Illegal characters in option name: {}'.format(value))
return value
def format(self, value):
return None if value is None else unicode(value)
class RegularExpression(Validator):
""" Validates regular expression option values.
"""
def __call__(self, value):
if value is None:
return None
try:
value = re.compile(unicode(value))
except re.error as error:
raise ValueError('{}: {}'.format(unicode(error).capitalize(), value))
return value
def format(self, value):
return None if value is None else value.pattern
class Set(Validator):
""" Validates set option values.
"""
def __init__(self, *args):
self.membership = set(args)
def __call__(self, value):
if value is None:
return None
value = unicode(value)
if value not in self.membership:
raise ValueError('Unrecognized value: {}'.format(value))
return value
def format(self, value):
return self.__call__(value)
__all__ = ['Boolean', 'Code', 'Duration', 'File', 'Integer', 'List', 'Map', 'RegularExpression', 'Set']
package/default/analytic_stories.conf
+1741
View File
File diff suppressed because it is too large Load Diff
package/default/analyticstories.conf
+5681
View File
File diff suppressed because it is too large Load Diff
package/default/app.conf
+30
View File
@@ -0,0 +1,30 @@
## Splunk app configuration file
[install]
is_configured = false
state = enabled
state_change_requires_restart = false
build = 25951
[triggers]
reload.analytic_stories = simple
reload.usage_searches = simple
reload.use_case_library = simple
reload.correlationsearches = simple
reload.analyticstories = simple
reload.governance = simple
reload.managed_configurations = simple
reload.postprocess = simple
reload.content-version = simple
[launcher]
author = Splunk
version = 3.19.0
description = Explore the Analytic Stories included with ES Content Updates.
[ui]
is_visible = true
label = ES Content Updates
[package]
id = DA-ESS-ContentUpdate
package/default/collections.conf
+55
View File
@@ -0,0 +1,55 @@
#############
# Automatically generated by generator.py in splunk/security_content
# On Date: 2021-04-02T17:00:03 UTC
# Author: Splunk Security Research
# Contact: research@splunk.com
#############
[api_call_by_user_baseline]
enforceTypes = false
replicate = false
[cloud_instances_enough_data]
enforceTypes = false
replicate = false
[previously_seen_cloud_api_calls_per_user_role]
enforceTypes = false
replicate = false
[previously_seen_cloud_compute_creations_by_user]
enforceTypes = false
replicate = false
[previously_seen_cloud_compute_images]
enforceTypes = false
replicate = false
[previously_seen_cloud_compute_instance_types]
enforceTypes = false
replicate = false
[previously_seen_cloud_instance_modifications_by_user]
enforceTypes = false
replicate = false
[previously_seen_cloud_provisioning_activity_sources]
enforceTypes = false
replicate = false
[previously_seen_cloud_regions]
enforceTypes = false
replicate = false
[previously_seen_running_windows_services]
enforceTypes = false
replicate = false
[previously_seen_users_console_logins]
enforceTypes = false
replicate = false
[zoom_first_time_child_process]
enforceTypes = false
replicate = false
package/default/commands.conf
+10
View File
@@ -0,0 +1,10 @@
[dnstwist]
filename = dnstwist.py
chunked = true
# run story functionality has been moved to: https://github.com/splunk/analytic_story_execution'
# [runstory]
# filename = runstory.py
# chunked = true
# is_risky = true
package/default/content-version.conf
+2
View File
@@ -0,0 +1,2 @@
[content-version]
version = 3.19.0
package/default/data/ui/nav/default.xml
+7
View File
@@ -0,0 +1,7 @@
<nav search_view="search" color="#65A637">
<view name="escu_summary" default="true"/>
<view name="feedback"/>
<view name="search"/>
<view name="escu_usage"/>
<a href="http://docs.splunk.com/Documentation/ESSOC">Docs</a>
</nav>
package/default/data/ui/panels/workbench_panel_all_backup_logs_for_host___response_task.xml
+9
View File
@@ -0,0 +1,9 @@
<panel>
<table>
<search>
<query>| search sourcetype="netbackup_logs" dest=$dest$</query>
</search>
<option name="drilldown">cell</option>
<option name="wrap">false</option>
</table>
</panel>
package/default/data/ui/panels/workbench_panel_amazon_eks_kubernetes_activity_by_src_ip___response_task.xml
+9
View File
@@ -0,0 +1,9 @@
<panel>
<table>
<search>
<query>sourcetype="aws:cloudwatchlogs:eks" |rename sourceIPs{} as src_ip |search src_ip=$src_ip$ | stats count min(_time) as firstTime max(_time) as lastTime values(user.username) values(requestURI) values(verb) values(userAgent) by source annotations.authorization.k8s.io/decision src_ip</query>
</search>
<option name="drilldown">cell</option>
<option name="wrap">false</option>
</table>
</panel>
package/default/data/ui/panels/workbench_panel_aws_investigate_security_hub_alerts_by_dest___response_task.xml
+9
View File
@@ -0,0 +1,9 @@
<panel>
<table>
<search>
<query>sourcetype="aws:securityhub:firehose" "findings{}.Resources{}.Type"=AWSEC2Instance | rex field=findings{}.Resources{}.Id .*instance/(?&lt;instance&gt;.*)| rename instance as dest| search dest = $dest$ |rename findings{}.* as * | rename Remediation.Recommendation.Text as Remediation | table dest Title ProductArn Description FirstObservedAt RecordState Remediation</query>
</search>
<option name="drilldown">cell</option>
<option name="wrap">false</option>
</table>
</panel>
package/default/data/ui/panels/workbench_panel_aws_investigate_user_activities_by_accesskeyid___response_task.xml
+9
View File
@@ -0,0 +1,9 @@
<panel>
<table>
<search>
<query>| search sourcetype=aws:cloudtrail | rename userIdentity.accessKeyId as accessKeyId| search accessKeyId=$accessKeyId$ | spath output=user path=userIdentity.arn | rename sourceIPAddress as src_ip | table _time, user, src_ip, awsRegion, eventName, errorCode, errorMessage</query>
</search>
<option name="drilldown">cell</option>
<option name="wrap">false</option>
</table>
</panel>
package/default/data/ui/panels/workbench_panel_aws_investigate_user_activities_by_arn___response_task.xml
+9
View File
@@ -0,0 +1,9 @@
<panel>
<table>
<search>
<query>| search sourcetype=aws:cloudtrail | search user=$user$| table _time userIdentity.type userIdentity.userName userIdentity.arn aws_account_id src awsRegion eventName eventType</query>
</search>
<option name="drilldown">cell</option>
<option name="wrap">false</option>
</table>
</panel>
package/default/data/ui/panels/workbench_panel_aws_network_acl_details_from_id___response_task.xml
+9
View File
@@ -0,0 +1,9 @@
<panel>
<table>
<search>
<query>| search sourcetype=aws:description| rename id as networkAclId | search networkAclId=$networkAclId$ | table id account_id vpc_id network_acl_entries{}.*</query>
</search>
<option name="drilldown">cell</option>
<option name="wrap">false</option>
</table>
</panel>
package/default/data/ui/panels/workbench_panel_aws_network_interface_details_via_resourceid___response_task.xml
+9
View File
@@ -0,0 +1,9 @@
<panel>
<table>
<search>
<query>| search sourcetype=aws:config resourceId=$resourceId$ | table _time ARN relationships{}.resourceType relationships{}.name relationships{}.resourceId configuration.privateIpAddresses{}.privateIpAddress configuration.privateIpAddresses{}.association.publicIp</query>
</search>
<option name="drilldown">cell</option>
<option name="wrap">false</option>
</table>
</panel>
package/default/data/ui/panels/workbench_panel_aws_s3_bucket_details_via_bucketname___response_task.xml
+9
View File
@@ -0,0 +1,9 @@
<panel>
<table>
<search>
<query>| search sourcetype=aws:config | rename resourceId as bucketName |search bucketName=$bucketName$ | table resourceCreationTime bucketName vendor_region action aws_account_id supplementaryConfiguration.AccessControlList</query>
</search>
<option name="drilldown">cell</option>
<option name="wrap">false</option>
</table>
</panel>
package/default/data/ui/panels/workbench_panel_gcp_kubernetes_activity_by_src_ip___response_task.xml
+9
View File
@@ -0,0 +1,9 @@
<panel>
<table>
<search>
<query>sourcetype="google:gcp:pubsub:message" | rename data.protoPayload.requestMetadata.callerIp as src_ip | search src_ip =$src_ip$ | stats count min(_time) as firstTime max(_time) as lastTime values(data.protoPayload.methodName) as method_names values(data.protoPayload.resourceName) as resource_name values(data.protoPayload.requestMetadata.callerSuppliedUserAgent) as http_user_agent values(data.protoPayload.authenticationInfo.principalEmail) as user values(data.protoPayload.status.message) by src_ip data.resource.labels.cluster_name data.resource.type</query>
</search>
<option name="drilldown">cell</option>
<option name="wrap">false</option>
</table>
</panel>
package/default/data/ui/panels/workbench_panel_get_all_aws_activity_from_city___response_task.xml
+9
View File
@@ -0,0 +1,9 @@
<panel>
<table>
<search>
<query>| search sourcetype=aws:cloudtrail | iplocation sourceIPAddress | search City=$City$ | spath output=user path=userIdentity.arn | spath output=awsUserName path=userIdentity.userName | spath output=userType path=userIdentity.type | rename sourceIPAddress as src_ip | table _time, City, user, userName, userType, src_ip, awsRegion, eventName, errorCode</query>
</search>
<option name="drilldown">cell</option>
<option name="wrap">false</option>
</table>
</panel>
package/default/data/ui/panels/workbench_panel_get_all_aws_activity_from_country___response_task.xml
+9
View File
@@ -0,0 +1,9 @@
<panel>
<table>
<search>
<query>| search sourcetype=aws:cloudtrail | iplocation sourceIPAddress | search Country=$Country$ | spath output=user path=userIdentity.arn | spath output=awsUserName path=userIdentity.userName | spath output=userType path=userIdentity.type | rename sourceIPAddress as src_ip | table _time, Country, user, userName, userType, src_ip, awsRegion, eventName, errorCode</query>
</search>
<option name="drilldown">cell</option>
<option name="wrap">false</option>
</table>
</panel>
package/default/data/ui/panels/workbench_panel_get_all_aws_activity_from_ip_address___response_task.xml
+9
View File
@@ -0,0 +1,9 @@
<panel>
<table>
<search>
<query>| search sourcetype=aws:cloudtrail | iplocation sourceIPAddress | search src_ip=$src_ip$ | spath output=user path=userIdentity.arn | spath output=awsUserName path=userIdentity.userName | spath output=userType path=userIdentity.type | rename sourceIPAddress as src_ip | table _time, user, userName, userType, src_ip, awsRegion, eventName, errorCode</query>
</search>
<option name="drilldown">cell</option>
<option name="wrap">false</option>
</table>
</panel>
package/default/data/ui/panels/workbench_panel_get_all_aws_activity_from_region___response_task.xml
+9
View File
@@ -0,0 +1,9 @@
<panel>
<table>
<search>
<query>| search sourcetype=aws:cloudtrail | iplocation sourceIPAddress | search Region=$Region$ | spath output=user path=userIdentity.arn | spath output=awsUserName path=userIdentity.userName | spath output=userType path=userIdentity.type | rename sourceIPAddress as src_ip | table _time, Region, user, userName, userType, src_ip, awsRegion, eventName, errorCode</query>
</search>
<option name="drilldown">cell</option>
<option name="wrap">false</option>
</table>
</panel>
package/default/data/ui/panels/workbench_panel_get_backup_logs_for_endpoint___response_task.xml
+9
View File
@@ -0,0 +1,9 @@
<panel>
<table>
<search>
<query>| search sourcetype="netbackup_logs" COMPUTERNAME=$dest$ | rename COMPUTERNAME as dest, MESSAGE as signature | table _time, dest, signature</query>
</search>
<option name="drilldown">cell</option>
<option name="wrap">false</option>
</table>
</panel>
package/default/data/ui/panels/workbench_panel_get_certificate_logs_for_a_domain___response_task.xml
+9
View File
@@ -0,0 +1,9 @@
<panel>
<table>
<search>
<query>| tstats `summariesonly` count min(_time) as firstTime max(_time) as lastTime FROM datamodel=Certificates.All_Certificates where All_Certificates.SSL.ssl_subject_common_name=*$domain$ by All_Certificates.dest All_Certificates.src All_Certificates.SSL.ssl_issuer_common_name All_Certificates.SSL.ssl_subject_common_name All_Certificates.SSL.ssl_hash | `drop_dm_object_name(All_Certificates)` | `drop_dm_object_name(SSL)` | rename ssl_subject_common_name as domain | `security_content_ctime(firstTime)` | `security_content_ctime(lastTime)`</query>
</search>
<option name="drilldown">cell</option>
<option name="wrap">false</option>
</table>
</panel>
package/default/data/ui/panels/workbench_panel_get_dns_server_history_for_a_host___response_task.xml
+9
View File
@@ -0,0 +1,9 @@
<panel>
<table>
<search>
<query>| search tag=dns src_ip=$src_ip$ dest_port=53 | streamstats time_window=1d count values(dest_ip) as dcip by src_ip | table date_mday src_ip dcip count | sort -count</query>
</search>
<option name="drilldown">cell</option>
<option name="wrap">false</option>
</table>
</panel>
package/default/data/ui/panels/workbench_panel_get_dns_traffic_ratio___response_task.xml
+9
View File
@@ -0,0 +1,9 @@
<panel>
<table>
<search>
<query>| tstats allow_old_summaries=true sum(All_Traffic.bytes_out) as "bytes_out" sum(All_Traffic.bytes_in) as "bytes_in" from datamodel=Network_Traffic where nodename=All_Traffic All_Traffic.dest_port=53 by All_Traffic.src All_Traffic.dest| `drop_dm_object_name(All_Traffic)` | rename src as src_ip | rename dest as dest_ip | search src_ip=$src_ip$ | search dest_ip = $dest_ip | eval ratio = (bytes_out/bytes_in) | table ratio</query>
</search>
<option name="drilldown">cell</option>
<option name="wrap">false</option>
</table>
</panel>
package/default/data/ui/panels/workbench_panel_get_ec2_instance_details_by_instanceid___response_task.xml
+9
View File
@@ -0,0 +1,9 @@
<panel>
<table>
<search>
<query>| search sourcetype="aws:description" source="*:ec2_instances"| dedup id sortby -_time |rename id as instanceId| search instanceId=$instanceId$ | spath output=tags path=tags | eval tags=mvzip(key,value," = "), ip_address=if((ip_address == "null"),private_ip_address,ip_address) | table id, tags.Name, aws_account_id, placement, instance_type, key_name, ip_address, launch_time, state, vpc_id, subnet_id, tags | rename aws_account_id as "Account ID", id as ID, instance_type as Type, ip_address as "IP Address", key_name as "Key Pair", launch_time as "Launch Time", placement as "Availability Zone", state as State, subnet_id as Subnet, "tags.Name" as Name, vpc_id as VPC</query>
</search>
<option name="drilldown">cell</option>
<option name="wrap">false</option>
</table>
</panel>
package/default/data/ui/panels/workbench_panel_get_ec2_launch_details___response_task.xml
+9
View File
@@ -0,0 +1,9 @@
<panel>
<table>
<search>
<query>| search sourcetype=aws:cloudtrail dest=$dest$ |rename userIdentity.arn as arn, responseElements.instancesSet.items{}.instanceId as dest, responseElements.instancesSet.items{}.privateIpAddress as privateIpAddress, responseElements.instancesSet.items{}.imageId as amiID, responseElements.instancesSet.items{}.architecture as architecture, responseElements.instancesSet.items{}.keyName as keyName | table arn, awsRegion, dest, architecture, privateIpAddress, amiID, keyName</query>
</search>
<option name="drilldown">cell</option>
<option name="wrap">false</option>
</table>
</panel>
package/default/data/ui/panels/workbench_panel_get_email_info___response_task.xml
+9
View File
@@ -0,0 +1,9 @@
<panel>
<table>
<search>
<query>| from datamodel Email.All_Email | search message_id=$message_id$</query>
</search>
<option name="drilldown">cell</option>
<option name="wrap">false</option>
</table>
</panel>
package/default/data/ui/panels/workbench_panel_get_emails_from_specific_sender___response_task.xml
+9
View File
@@ -0,0 +1,9 @@
<panel>
<table>
<search>
<query>| from datamodel Email.All_Email | search src_user=$src_user$</query>
</search>
<option name="drilldown">cell</option>
<option name="wrap">false</option>
</table>
</panel>
package/default/data/ui/panels/workbench_panel_get_first_occurrence_and_last_occurrence_of_a_mac_address___response_task.xml
+9
View File
@@ -0,0 +1,9 @@
<panel>
<table>
<search>
<query>| tstats `security_content_summariesonly` count min(_time) as firstTime max(_time) as lastTime from datamodel=Network_Sessions where nodename=All_Sessions.DHCP All_Sessions.signature=DHCPREQUEST All_Sessions.All_Sessions.src_mac= $src_mac$ by All_Sessions.src_ip All_Sessions.user | `security_content_ctime(lastTime)` | `security_content_ctime(firstTime)`</query>
</search>
<option name="drilldown">cell</option>
<option name="wrap">false</option>
</table>
</panel>
package/default/data/ui/panels/workbench_panel_get_history_of_email_sources___response_task.xml
+9
View File
@@ -0,0 +1,9 @@
<panel>
<table>
<search>
<query>|tstats `security_content_summariesonly` values(All_Email.dest) as dest values(All_Email.recipient) as recepient min(_time) as firstTime max(_time) as lastTime count from datamodel=Email.All_Email by All_Email.src |`drop_dm_object_name(All_Email)` | `security_content_ctime(firstTime)` | `security_content_ctime(lastTime)` | search src=$src$</query>
</search>
<option name="drilldown">cell</option>
<option name="wrap">false</option>
</table>
</panel>
package/default/data/ui/panels/workbench_panel_get_logon_rights_modifications_for_endpoint___response_task.xml
+9
View File
@@ -0,0 +1,9 @@
<panel>
<table>
<search>
<query>| search eventtype=wineventlog_security (signature_id=4718 OR signature_id=4717) dest=$dest$ | rename user as "Account Modified" | table _time, dest, "Account Modified", Access_Right, signature</query>
</search>
<option name="drilldown">cell</option>
<option name="wrap">false</option>
</table>
</panel>
package/default/data/ui/panels/workbench_panel_get_logon_rights_modifications_for_user___response_task.xml
+9
View File
@@ -0,0 +1,9 @@
<panel>
<table>
<search>
<query>| search eventtype=wineventlog_security (signature_id=4718 OR signature_id=4717) user=$user$ | rename user as "Account Modified" | table _time, dest, "Account Modified", Access_Right, signature</query>
</search>
<option name="drilldown">cell</option>
<option name="wrap">false</option>
</table>
</panel>
package/default/data/ui/panels/workbench_panel_get_notable_history___response_task.xml
+9
View File
@@ -0,0 +1,9 @@
<panel>
<table>
<search>
<query>| search `notable` | search dest=$dest$ | table _time, dest, rule_name, owner, priority, severity, status_description</query>
</search>
<option name="drilldown">cell</option>
<option name="wrap">false</option>
</table>
</panel>
package/default/data/ui/panels/workbench_panel_get_outbound_emails_to_hidden_cobra_threat_actors___response_task.xml
+9
View File
@@ -0,0 +1,9 @@
<panel>
<table>
<search>
<query>| from datamodel Email.All_Email | search recipient=misswang8107@gmail.com OR src_user=redhat@gmail.com | stats count earliest(_time) as firstTime, latest(_time) as lastTime values(dest) values(src) by src_user recipient | `security_content_ctime(firstTime)` | `security_content_ctime(lastTime)`</query>
</search>
<option name="drilldown">cell</option>
<option name="wrap">false</option>
</table>
</panel>
package/default/data/ui/panels/workbench_panel_get_parent_process_info___response_task.xml
+9
View File
@@ -0,0 +1,9 @@
<panel>
<table>
<search>
<query>| tstats `summariesonly` count values(Processes.process) as process min(_time) as firstTime max(_time) as lastTime FROM datamodel=Endpoint.Processes by Processes.user Processes.parent_process_name Processes.process_name Processes.dest | `drop_dm_object_name("Processes")` | search parent_process_name= $parent_process_name$ |search dest = $dest$ | `security_content_ctime(firstTime)` | `security_content_ctime(lastTime)`</query>
</search>
<option name="drilldown">cell</option>
<option name="wrap">false</option>
</table>
</panel>
package/default/data/ui/panels/workbench_panel_get_process_file_activity___response_task.xml
+9
View File
@@ -0,0 +1,9 @@
<panel>
<table>
<search>
<query>| tstats `security_content_summariesonly` values(Filesystem.file_name) as file_name values(Filesystem.dest) as dest, values(Filesystem.process_name) as process_name from datamodel=Endpoint.Filesystem by Filesystem.dest Filesystem.process_name Filesystem.file_path, Filesystem.action, _time | `drop_dm_object_name(Filesystem)` | search dest=$dest$ | search process_name=$process_name$ | table _time, process_name, dest, action, file_name, file_path</query>
</search>
<option name="drilldown">cell</option>
<option name="wrap">false</option>
</table>
</panel>
package/default/data/ui/panels/workbench_panel_get_process_info___response_task.xml
+9
View File
@@ -0,0 +1,9 @@
<panel>
<table>
<search>
<query>| tstats `summariesonly` count values(Processes.process) as process min(_time) as firstTime max(_time) as lastTime FROM datamodel=Endpoint.Processes by Processes.user Processes.parent_process_name Processes.process_name Processes.dest | `drop_dm_object_name("Processes")` | search process_name= $process_name$ | search dest = $dest$ | `security_content_ctime(firstTime)` | `security_content_ctime(lastTime)`</query>
</search>
<option name="drilldown">cell</option>
<option name="wrap">false</option>
</table>
</panel>
package/default/data/ui/panels/workbench_panel_get_process_information_for_port_activity___response_task.xml
+9
View File
@@ -0,0 +1,9 @@
<panel>
<table>
<search>
<query>| tstats `security_content_summariesonly` count min(_time) max(_time) as lastTime from datamodel=Endpoint.Processes by Processes.process_name Processes.user Processes.dest Processes.process_id | `drop_dm_object_name(Processes)` | `security_content_ctime(firstTime)` | `security_content_ctime(lastTime)` | search dest=$dest$ | join dest type=inner [| tstats `security_content_summariesonly` count from datamodel=Endpoint.Ports by Ports.process_id Ports.src Ports.dest_port | `drop_dm_object_name(Ports)` | search dest_port=$dest_port$ | rename src as dest]</query>
</search>
<option name="drilldown">cell</option>
<option name="wrap">false</option>
</table>
</panel>
package/default/data/ui/panels/workbench_panel_get_process_responsible_for_the_dns_traffic___response_task.xml
+9
View File
@@ -0,0 +1,9 @@
<panel>
<table>
<search>
<query>| tstats `security_content_summariesonly` count min(_time) max(_time) as lastTime from datamodel=Endpoint.Processes by Processes.parent_process Processes.process_name Processes.user Processes.dest Processes.process_id | `drop_dm_object_name(Processes)` | `security_content_ctime(firstTime)` | `security_content_ctime(lastTime)` | search dest = $dest$ | join dest type=inner [| tstats `security_content_summariesonly` count from datamodel=Endpoint.Ports where Ports.dest_port=53 by Ports.process_id Ports.src | `drop_dm_object_name(Ports)` | rename src as dest]</query>
</search>
<option name="drilldown">cell</option>
<option name="wrap">false</option>
</table>
</panel>
package/default/data/ui/panels/workbench_panel_get_sysmon_wmi_activity_for_host___response_task.xml
+9
View File
@@ -0,0 +1,9 @@
<panel>
<table>
<search>
<query>sourcetype="XmlWinEventLog:Microsoft-Windows-Sysmon/Operational" EventCode&gt;18 EventCode&lt;22 | rename host as dest | search dest=$dest$| table _time, dest, user, Name, Operation, EventType, Type, Query, Consumer, Filter</query>
</search>
<option name="drilldown">cell</option>
<option name="wrap">false</option>
</table>
</panel>
package/default/data/ui/panels/workbench_panel_get_web_session_information_via_session_id___response_task.xml
+9
View File
@@ -0,0 +1,9 @@
<panel>
<table>
<search>
<query>| search sourcetype=stream:http session_id = $session_id$ | stats values(url) values(http_user_agent) by src_ip status</query>
</search>
<option name="drilldown">cell</option>
<option name="wrap">false</option>
</table>
</panel>
package/default/data/ui/panels/workbench_panel_investigate_aws_activities_via_region_name___response_task.xml
+9
View File
@@ -0,0 +1,9 @@
<panel>
<table>
<search>
<query>| search sourcetype=aws:cloudtrail vendor_region=$vendor_region$| rename requestParameters.instancesSet.items{}.instanceId as instanceId | stats values(eventName) by user instanceId vendor_region</query>
</search>
<option name="drilldown">cell</option>
<option name="wrap">false</option>
</table>
</panel>
package/default/data/ui/panels/workbench_panel_investigate_aws_user_activities_by_user_field___response_task.xml
+9
View File
@@ -0,0 +1,9 @@
<panel>
<table>
<search>
<query>| search sourcetype=aws:cloudtrail user=$user$ | table _time userIdentity.type userIdentity.userName userIdentity.arn aws_account_id src awsRegion eventName eventType </query>
</search>
<option name="drilldown">cell</option>
<option name="wrap">false</option>
</table>
</panel>
package/default/data/ui/panels/workbench_panel_investigate_failed_logins_for_multiple_destinations___response_task.xml
+9
View File
@@ -0,0 +1,9 @@
<panel>
<table>
<search>
<query>| tstats count `security_content_summariesonly` earliest(_time) as first_login latest(_time) as last_login dc(Authentication.dest) AS distinct_count_dest values(Authentication.dest) AS Authentication.dest values(Authentication.app) AS Authentication.app from datamodel=Authentication where Authentication.action=failure by Authentication.user | where distinct_count_dest &gt; 1 | `security_content_ctime(first_login)` | `security_content_ctime(last_login)` | `drop_dm_object_name("Authentication")` | search user=$user$</query>
</search>
<option name="drilldown">cell</option>
<option name="wrap">false</option>
</table>
</panel>
package/default/data/ui/panels/workbench_panel_investigate_network_traffic_from_src_ip___response_task.xml
+9
View File
@@ -0,0 +1,9 @@
<panel>
<table>
<search>
<query>| from datamodel Network_Traffic.All_Traffic | search src_ip=$src_ip$</query>
</search>
<option name="drilldown">cell</option>
<option name="wrap">false</option>
</table>
</panel>
package/default/data/ui/panels/workbench_panel_investigate_okta_activity_by_app___response_task.xml
+9
View File
@@ -0,0 +1,9 @@
<panel>
<table>
<search>
<query>eventtype=okta_log app=$app$ | rename client.geographicalContext.country as country, client.geographicalContext.state as state, client.geographicalContext.city as city | table _time, user, displayMessage, app, src_ip, state, city, result, outcome.reason</query>
</search>
<option name="drilldown">cell</option>
<option name="wrap">false</option>
</table>
</panel>
package/default/data/ui/panels/workbench_panel_investigate_okta_activity_by_ip_address___response_task.xml
+9
View File
@@ -0,0 +1,9 @@
<panel>
<table>
<search>
<query>eventtype=okta_log src_ip={src_ip} | rename client.geographicalContext.country as country, client.geographicalContext.state as state, client.geographicalContext.city as city | table _time, user, displayMessage, app, src_ip, state, city, result, outcome.reason</query>
</search>
<option name="drilldown">cell</option>
<option name="wrap">false</option>
</table>
</panel>
package/default/data/ui/panels/workbench_panel_investigate_pass_the_hash_attempts___response_task.xml
+9
View File
@@ -0,0 +1,9 @@
<panel>
<table>
<search>
<query>`wineventlog_security` EventCode=4624 Logon_Type=9 AuthenticationPackageName=Negotiate | stats count earliest(_time) as first_login latest(_time) as last_login by src_user dest | `security_content_ctime(first_login)` | `security_content_ctime(last_login)` | search dest=$dest$</query>
</search>
<option name="drilldown">cell</option>
<option name="wrap">false</option>
</table>
</panel>
package/default/data/ui/panels/workbench_panel_investigate_pass_the_ticket_attempts___response_task.xml
+9
View File
@@ -0,0 +1,9 @@
<panel>
<table>
<search>
<query>`wineventlog_security` EventCode=4768 OR EventCode=4769 | rex field=user "(?&lt;new_user&gt;[^\@]+)" | stats count BY new_user, dest, EventCode | stats max(count) AS max_count sum(count) AS sum_count BY new_user, dest| search dest=$dest$ | where sum_count/max_count!=2 | rename new_user AS user </query>
</search>
<option name="drilldown">cell</option>
<option name="wrap">false</option>
</table>
</panel>
package/default/data/ui/panels/workbench_panel_investigate_previous_unseen_user___response_task.xml
+9
View File
@@ -0,0 +1,9 @@
<panel>
<table>
<search>
<query>| tstats count `security_content_summariesonly` earliest(_time) as first_login latest(_time) as last_login values(Authentication.dest) AS Authentication.dest values(Authentication.app) AS Authentication.app values(Authentication.action) AS Authentication.action from datamodel=Authentication where Authentication.action=success by _time, Authentication.user | bucket _time span=30d | stats count min(first_login) as first_login max(last_login) as last_login values(Authentication.dest) AS Authentication.dest by Authentication.user | where count=1 | where first_login &gt;= relative_time(now(), "-30d") | `security_content_ctime(first_login)` | `security_content_ctime(last_login)` | `drop_dm_object_name("Authentication")` | search dest=$dest$</query>
</search>
<option name="drilldown">cell</option>
<option name="wrap">false</option>
</table>
</panel>
package/default/data/ui/panels/workbench_panel_investigate_successful_remote_desktop_authentications___response_task.xml
+9
View File
@@ -0,0 +1,9 @@
<panel>
<table>
<search>
<query>| tstats `security_content_summariesonly` count min(_time) as firstTime max(_time) as lastTime from datamodel=Authentication where Authentication.signature_id=4624 Authentication.app=win:remote by Authentication.src Authentication.dest Authentication.app Authentication.user Authentication.signature Authentication.src_nt_domain | `security_content_ctime(lastTime)` | `security_content_ctime(firstTime)` | `drop_dm_object_name("Authentication")` | search dest=$dest$ | table firstTime lastTime src src_nt_domain dest user app count | sort count</query>
</search>
<option name="drilldown">cell</option>
<option name="wrap">false</option>
</table>
</panel>
package/default/data/ui/panels/workbench_panel_investigate_suspicious_strings_in_http_header___response_task.xml
+9
View File
@@ -0,0 +1,9 @@
<panel>
<table>
<search>
<query>| search sourcetype=stream:http | search src_ip=$src_ip$ | search dest_ip=$dest_ip$ | eval cs_content_type_length = len(cs_content_type) | search cs_content_type_length &gt; 100 | rex field="cs_content_type" (?&lt;suspicious_strings&gt;cmd.exe) | eval suspicious_strings_found=if(match(cs_content_type, "application"), "True", "False") | rename suspicious_strings_found AS "Suspicious Content-Type Found" | fields "Suspicious Content-Type Found", dest_ip, src_ip, suspicious_strings, cs_content_type, cs_content_type_length, url</query>
</search>
<option name="drilldown">cell</option>
<option name="wrap">false</option>
</table>
</panel>
package/default/data/ui/panels/workbench_panel_investigate_user_activities_in_okta___response_task.xml
+9
View File
@@ -0,0 +1,9 @@
<panel>
<table>
<search>
<query>eventtype=okta_log user=$user$ | rename client.geographicalContext.country as country, client.geographicalContext.state as state, client.geographicalContext.city as city | table _time, user, displayMessage, app, src_ip, state, city, result, outcome.reason</query>
</search>
<option name="drilldown">cell</option>
<option name="wrap">false</option>
</table>
</panel>

Some files were not shown because too many files have changed in this diff Show More