admin/splunk-security_content
1
0
Fork 0
mirror of https://github.com/splunk/security_content synced 2026-06-08 17:32:49 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'driverload' of https://github.com/splunk/security_content into driverload

Browse Source
This commit is contained in:
mhaag-spl
2022-05-16 14:04:32 -06:00
parent a071bdbd22 938dd6824b
commit e575a2f5bf
78 changed files with 379 additions and 12653 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/pull_request_template.md
+14 -8
View File
@@ -1,12 +1,18 @@
# PR Template for new Detections
For Authors:
- [ ] Make sure that CI/CD [detection-testing and build-and-validate](https://github.com/splunk/security_content/actions) jobs passed ✔️.
### Details
*_what does this PR have in it, screenshots are nice 😄_
### Author Checklist
For Reviewers:
- [ ] Verify CI/CD jobs have passed without errors.
- [ ] Validate SPL logic.
- [ ] Validate tags, description, and how to implement.
- [ ] Validate name matches `<platform>_<mitre att&ck technique>_<short description>`
- [ ] Verify references match analytic.
- [ ] Is there an Atomic Test?
- [ ] Make sure that CI/CD [detection-testing and build-and-validate](https://github.com/splunk/security_content/actions) jobs passed ✔️
- [ ] Is there an Atomic Test? Is GUID on the test file under array: `atomic_test_guid`, [example]()?
### Review Checklist
- [ ] Validated SPL logic.
- [ ] Validated tags, description, and how to implement.
- [ ] Verified references match analytic.
bin/contentctl_project/contentctl_core/tests/application/use_cases/test_data_input/detections/endpoint/cobalt_strike_named_pipes.yml
-89
View File
@@ -1,89 +0,0 @@
name: Cobalt Strike Named Pipes
id: 5876d429-0240-4709-8b93-ea8330b411b5
version: 2
date: '2022-05-04'
author: Michael Haag, Splunk
type: TTP
datamodel: []
description: 'The following analytic identifies the use of default or publicly known
named pipes used with Cobalt Strike. A named pipe is a named, one-way or duplex
pipe for communication between the pipe server and one or more pipe clients. Cobalt
Strike uses named pipes in many ways and has default values used with the Artifact
Kit and Malleable C2 Profiles. The following query assists with identifying these
default named pipes. Each EDR product presents named pipes a little different. Consider
taking the values and generating a query based on the product of choice. \
Upon triage, review the process performing the named pipe. If it is explorer.exe,
It is possible it was injected into by another process. Review recent parallel processes
to identify suspicious patterns or behaviors. A parallel process may have a network
connection, review and follow the connection back to identify any file modifications.'
search: '`sysmon` EventID=17 OR EventID=18 PipeName IN (\\msagent_*, \\DserNamePipe*,
\\srvsvc_*, \\postex_*, \\status_*, \\MSSE-*, \\spoolss_*, \\win_svc*,
\\ntsvcs*, \\winsock*, \\UIA_PIPE*) | stats count min(_time) as firstTime max(_time)
as lastTime by Computer, process_name, process_id process_path, PipeName | rename
Computer as dest | `security_content_ctime(firstTime)` | `security_content_ctime(lastTime)`
| `cobalt_strike_named_pipes_filter`'
how_to_implement: To successfully implement this search, you need to be ingesting
logs with the process name, parent process, and command-line executions from your
endpoints. If you are using Sysmon, you must have at least version 6.0.4 of the
Sysmon TA.
known_false_positives: The idea of using named pipes with Cobalt Strike is to blend
in. Therefore, some of the named pipes identified and added may cause false positives.
Filter by process name or pipe name to reduce false positives.
references:
- https://attack.mitre.org/techniques/T1218/009/
- https://docs.microsoft.com/en-us/windows/win32/ipc/named-pipes
- https://gist.github.com/MHaggis/6c600e524045a6d49c35291a21e10752
- https://www.fireeye.com/blog/threat-research/2021/05/shining-a-light-on-darkside-ransomware-operations.html
tags:
analytic_story:
- Cobalt Strike
- Trickbot
- DarkSide Ransomware
asset_type: Endpoint
automated_detection_testing: passed
cis20:
- CIS 8
confidence: 90
context:
- Source:Endpoint
- Stage:Defense Evasion
dataset:
- https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1055/cobalt_strike/windows-sysmon.log
impact: 80
kill_chain_phases:
- Actions on Objectives
message: An instance of $process_name$ was identified on endpoint $Computer$ by
user $user$ accessing known suspicious named pipes related to Cobalt Strike.
mitre_attack_id:
- T1055
nist:
- PR.PT
- DE.CM
observable:
- name: user
type: User
role:
- Victim
- name: Computer
type: Hostname
role:
- Victim
- name: process_name
type: Process
role:
- Parent Process
product:
- Splunk Enterprise
- Splunk Enterprise Security
- Splunk Cloud
required_fields:
- _time
- EventID
- PipeName
- Computer
- process_name
- process_path
- process_id
risk_score: 72
security_domain: endpoint
bin/contentctl_project/contentctl_infrastructure/adapter/templates/transforms.j2
+10 -10
View File
@@ -1,34 +1,34 @@
{% for lookup in objects %}
[{{ lookup.name }}]
{% if lookup.filename is defined %}
filename = {{ lookup.filename }}
{% if lookup.filename is defined and lookup.filename != None %}
filename = {{ lookup.filename }}
{% else %}
collection = {{ lookup.collection }}
external_type = kvstore
{% endif %}
{% if lookup.default_match is defined %}
{% if lookup.default_match is defined and lookup.default_match != None %}
default_match = {{ lookup.default_match }}
{% endif %}
{% if lookup.case_sensitive_match is defined %}
{% if lookup.case_sensitive_match is defined and lookup.case_sensitive_match != None %}
case_sensitive_match = {{ lookup.case_sensitive_match }}
{% endif %}
{% if lookup.description is defined %}
{% if lookup.description is defined and lookup.description != None %}
# description = {{ lookup.description }}
{% endif %}
{% if lookup.match_type is defined %}
{% if lookup.match_type is defined and lookup.match_type != None %}
match_type = {{ lookup.match_type }}
{% endif %}
{% if lookup.max_matches is defined %}
{% if lookup.max_matches is defined and lookup.max_matches != None %}
max_matches = {{ lookup.max_matches }}
{% endif %}
{% if lookup.min_matches is defined %}
{% if lookup.min_matches is defined and lookup.min_matches != None %}
min_matches = {{ lookup.min_matches }}
{% endif %}
{% if lookup.fields_list is defined %}
{% if lookup.fields_list is defined and lookup.fields_list != None %}
fields_list = {{ lookup.fields_list }}
{% endif %}
{% if lookup.filter is defined %}
{% if lookup.filter is defined and lookup.filter != None %}
filter = {{ lookup.filter }}
{% endif %}
bin/docker_detection_tester/modules/validate_args.py
+9 -1
View File
@@ -77,7 +77,15 @@ setup_schema = {
"app_version": None,
"local_path": None
},
#The default apps below were taken from the attack_range loadout: https://github.com/splunk/attack_range/blob/develop/attack_range.conf.template
"PALO_ALTO_NETWORKS_ADD_ON_FOR_SPLUNK": {
"app_number": 2757,
"app_version": "7.1.0",
"http_path": "https://attack-range-appbinaries.s3.us-west-2.amazonaws.com/palo-alto-networks-add-on-for-splunk_710.tgz"
},
"SPLUNK_ADD_ON_FOR_MICROSOFT_WINDOWS": {
"app_number": 742,
"app_version": "8.4.0",
@@ -343,4 +351,4 @@ def validate(configuration: dict, skip_password_accessibility_check:bool=True) -
except Exception as e:
print("There was an error validation the configuration: [%s]" % (
str(e)), file=sys.stderr)
return None, setup_schema
return None, setup_schema
bin/docker_detection_tester/test_config_github_actions.json
+6 -1
View File
@@ -1,5 +1,10 @@
{
"apps": {
"PALO_ALTO_NETWORKS_ADD_ON_FOR_SPLUNK": {
"app_number": 2757,
"app_version": "7.1.0",
"http_path": "https://attack-range-appbinaries.s3.us-west-2.amazonaws.com/palo-alto-networks-add-on-for-splunk_710.tgz"
},
"ADD_ON_FOR_LINUX_SYSMON": {
"app_number": 6176,
"app_version": "1.0.4",
@@ -106,4 +111,4 @@
"Hunting",
"TTP"
]
}
}
detections/cloud/detect_aws_console_login_by_new_user.yml
+4 -4
View File
@@ -1,7 +1,7 @@
name: Detect AWS Console Login by New User
id: bc91a8cd-35e7-4bb2-6140-e756cc46fd71
version: 1
date: '2020-05-28'
version: 2
date: '2022-05-10'
author: Rico Valdez, Splunk
type: Hunting
datamodel:
@@ -21,8 +21,8 @@ search: '| tstats earliest(_time) as firstTime latest(_time) as lastTime from da
how_to_implement: You must install and configure the Splunk Add-on for AWS (version
5.1.0 or later) and Enterprise Security 6.2, which contains the required updates
to the Authentication data model for cloud use cases. Run the `Previously Seen Users
in AWS CloudTrail - Initial` support search only once to create a baseline of previously
seen IAM users within the last 30 days. Run `Previously Seen Users in AWS CloudTrail
in CloudTrail - Initial` support search only once to create a baseline of previously
seen IAM users within the last 30 days. Run `Previously Seen Users in CloudTrail
- Update` hourly (or more frequently depending on how often you run the detection
searches) to refresh the baselines.
known_false_positives: When a legitimate new user logins for the first time, this
detections/deprecated/monitor_dns_for_brand_abuse.yml
+1 -1
View File
@@ -17,7 +17,7 @@ how_to_implement: You need to ingest data from your DNS logs. Specifically you m
This approach allows you to also create your own localized passive DNS capability
which can aid you in future investigations. You also need to have run the search
"ESCU - DNSTwist Domain Names", which creates the permutations of the domain that
will be checked for.
will be checked for. You also need the [`dnstwist`](https://gist.github.com/d1vious/c4c2aae7fa7d5cbb1f24adc5f6303) custom command.
known_false_positives: None at this time
references: []
tags:
detections/endpoint/get_addefaultdomainpasswordpolicy_with_powershell_script_block.yml
+7 -6
View File
@@ -1,7 +1,7 @@
name: Get ADDefaultDomainPasswordPolicy with Powershell Script Block
id: 1ff7ccc8-065a-11ec-91e4-acde48001122
version: 1
date: '2021-08-26'
version: 2
date: '2022-03-22'
author: Teoderick Contreras, Mauricio Velazco, Splunk
type: Hunting
datamodel: []
@@ -10,9 +10,10 @@ description: The following analytic utilizes PowerShell Script Block Logging (Ev
used to obtain the password policy in a Windows domain. Red Teams and adversaries
alike may use PowerShell to enumerate domain policies for situational awareness
and Active Directory Discovery.
search: '`powershell` EventCode=4104 Message ="*Get-ADDefaultDomainPasswordPolicy*"
| stats count min(_time) as firstTime max(_time) as lastTime by EventCode Message
ComputerName User | `security_content_ctime(firstTime)` | `security_content_ctime(lastTime)`
search: '`powershell` EventCode=4104 ScriptBlockText ="*Get-ADDefaultDomainPasswordPolicy*"
| stats count min(_time) as firstTime max(_time) as lastTime by EventCode ScriptBlockText Computer user_id
| `security_content_ctime(firstTime)`
| `security_content_ctime(lastTime)`
| `get_addefaultdomainpasswordpolicy_with_powershell_script_block_filter`'
how_to_implement: The following Hunting analytic requires PowerShell operational logs
to be imported. Modify the powershell macro as needed to match the sourcetype or
@@ -30,7 +31,7 @@ tags:
- Source:Endpoint
- Stage:Discovery
dataset:
- https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1201/pwd_policy_discovery/windows-powershell.log
- https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1201/pwd_policy_discovery/windows-powershell-xml.log
impact: 30
kill_chain_phases:
- Reconnaissance
detections/endpoint/get_domainuser_with_powershell_script_block.yml
+8 -6
View File
@@ -1,7 +1,7 @@
name: Get DomainUser with PowerShell Script Block
id: 61994268-04f4-11ec-865c-acde48001122
version: 1
date: '2021-08-24'
version: 2
date: '2022-03-22'
author: Teoderick Contreras, Mauricio Velazco, Splunk
type: TTP
datamodel: []
@@ -10,9 +10,11 @@ description: The following analytic utilizes PowerShell Script Block Logging (Ev
part of PowerView, a PowerShell tool used to perform enumeration on Windows domains.
Red Teams and adversaries alike may use PowerView to enumerate domain users for
situational awareness and Active Directory Discovery.
search: '`powershell` EventCode=4104 Message = "*Get-DomainUser*" | stats count min(_time)
as firstTime max(_time) as lastTime by EventCode Message ComputerName User | `security_content_ctime(firstTime)`
| `security_content_ctime(lastTime)` | `get_domainuser_with_powershell_script_block_filter`'
search: '`powershell` EventCode=4104 ScriptBlockText = "*Get-DomainUser*"
| stats count min(_time) as firstTime max(_time) as lastTime by EventCode ScriptBlockText Computer user_id
| `security_content_ctime(firstTime)`
| `security_content_ctime(lastTime)`
| `get_domainuser_with_powershell_script_block_filter`'
how_to_implement: The following Hunting analytic requires PowerShell operational logs
to be imported. Modify the powershell macro as needed to match the sourcetype or
add index. This analytic is specific to 4104, or PowerShell Script Block Logging.
@@ -27,7 +29,7 @@ tags:
- Source:Endpoint
- Stage:Discovery
dataset:
- https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1087.002/AD_discovery/windows-powershell.log
- https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1087.002/AD_discovery/windows-powershell-xml.log
impact: 50
kill_chain_phases:
- Reconnaissance
detections/endpoint/get_foresttrust_with_powershell_script_block.yml
+7 -5
View File
@@ -17,13 +17,15 @@ description: 'The following analytic utilizes PowerShell Script Block Logging (E
During triage, review parallel processes using an EDR product or 4688 events. It
will be important to understand the timeline of events around this activity. Review
the entire logged PowerShell script block.'
search: '`powershell` EventCode=4104 Message = "*get-foresttrust*" | stats count min(_time)
as firstTime max(_time) as lastTime by Message OpCode ComputerName User EventCode
| `security_content_ctime(firstTime)` | `security_content_ctime(lastTime)` | `get_foresttrust_with_powershell_script_block_filter`'
search: '`powershell` EventCode=4104 ScriptBlockText = "*get-foresttrust*"
| stats count min(_time) as firstTime max(_time) as lastTime by EventCode ScriptBlockText Computer user_id
| `security_content_ctime(firstTime)`
| `security_content_ctime(lastTime)`
| `get_foresttrust_with_powershell_script_block_filter`'
how_to_implement: To successfully implement this analytic, you will need to enable
PowerShell Script Block Logging on some or all endpoints. Additional setup here
https://docs.splunk.com/Documentation/UBA/5.0.4.1/GetDataIn/AddPowerShell#Configure_module_logging_for_PowerShell.
known_false_positives: UPDATE_KNOWN_FALSE_POSITIVES
known_false_positives: False positives may be present. Tune as needed.
references:
- https://powersploit.readthedocs.io/en/latest/Recon/Get-ForestTrust/
tags:
@@ -34,7 +36,7 @@ tags:
- Source:Endpoint
- Stage:Discovery
dataset:
- https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1482/discovery/windows-powershell.log
- https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1482/discovery/windows-powershell-xml.log
impact: 30
kill_chain_phases:
- Reconnaissance
detections/endpoint/get_wmiobject_group_discovery_with_script_block_logging.yml
+7 -6
View File
@@ -1,7 +1,7 @@
name: Get WMIObject Group Discovery with Script Block Logging
id: 69df7f7c-155d-11ec-a055-acde48001122
version: 1
date: '2021-09-14'
version: 2
date: '2022-03-22'
author: Michael Haag, Splunk
type: Hunting
datamodel: []
@@ -18,9 +18,10 @@ description: 'The following analytic utilizes PowerShell Script Block Logging (E
During triage, review parallel processes using an EDR product or 4688 events. It
will be important to understand the timeline of events around this activity. Review
the entire logged PowerShell script block.'
search: '`powershell` EventCode=4104 Message = "*Get-WMIObject*" AND Message = "*Win32_Group*"
| stats count min(_time) as firstTime max(_time) as lastTime by Message OpCode
ComputerName User EventCode| `security_content_ctime(firstTime)` | `security_content_ctime(lastTime)`
search: '`powershell` EventCode=4104 ScriptBlockText = "*Get-WMIObject*" AND ScriptBlockText = "*Win32_Group*"
| stats count min(_time) as firstTime max(_time) as lastTime by EventCode ScriptBlockText Computer user_id
| `security_content_ctime(firstTime)`
| `security_content_ctime(lastTime)`
| `get_wmiobject_group_discovery_with_script_block_logging_filter`'
how_to_implement: To successfully implement this analytic, you will need to enable
PowerShell Script Block Logging on some or all endpoints. Additional setup here
@@ -41,7 +42,7 @@ tags:
- Source:Endpoint
- Stage:Discovery
dataset:
- https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1069.001/atomic_red_team/windows-powershell.log
- https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1069.001/atomic_red_team/windows-powershell-xml.log
impact: 30
kill_chain_phases:
- Reconnaissance
detections/endpoint/getadgroup_with_powershell_script_block.yml
+7 -5
View File
@@ -1,7 +1,7 @@
name: GetAdGroup with PowerShell Script Block
id: e4c73d68-794b-468d-b4d0-dac1772bbae7
version: 1
date: '2021-08-25'
version: 2
date: '2022-03-22'
author: Mauricio Velazco, Splunk
type: Hunting
datamodel: []
@@ -10,8 +10,10 @@ description: The following analytic utilizes PowerShell Script Block Logging (Ev
is used to return a list of all domain groups. Red Teams and adversaries may leverage
this commandlet to enumerate domain groups for situational awareness and Active
Directory Discovery.
search: '`powershell` EventCode=4104 (Message = "*Get-ADGroup*") | stats count min(_time)
as firstTime max(_time) as lastTime by EventCode Message ComputerName User | `security_content_ctime(firstTime)`
search: '`powershell` EventCode=4104 ScriptBlockText = "*Get-ADGroup*"
| stats count min(_time) as firstTime max(_time) as lastTime by EventCode ScriptBlockText Computer user_id
| `security_content_ctime(firstTime)`
| `security_content_ctime(lastTime)`
| `getadgroup_with_powershell_script_block_filter`'
how_to_implement: To successfully implement this analytic, you will need to enable
PowerShell Script Block Logging on some or all endpoints. Additional setup here
@@ -29,7 +31,7 @@ tags:
- Source:Endpoint
- Stage:Discovery
dataset:
- https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1069.002/AD_discovery/windows-powershell.log
- https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1069.002/AD_discovery/windows-powershell-xml.log
impact: 30
kill_chain_phases:
- Reconnaissance
detections/endpoint/getcurrent_user_with_powershell_script_block.yml
+7 -6
View File
@@ -1,7 +1,7 @@
name: GetCurrent User with PowerShell Script Block
id: 80879283-c30f-44f7-8471-d1381f6d437a
version: 1
date: '2021-09-13'
version: 2
date: '2022-03-22'
author: Mauricio Velazco, Splunk
type: Hunting
datamodel: []
@@ -10,9 +10,10 @@ description: The following analytic utilizes PowerShell Script Block Logging (Ev
class. This method returns an object that represents the current Windows user. Red
Teams and adversaries may leverage this method to identify the logged user on a
compromised endpoint for situational awareness and Active Directory Discovery.
search: '`powershell` EventCode=4104 (Message = "*[System.Security.Principal.WindowsIdentity]*"
AND Message = "*GetCurrent()*") | stats count min(_time) as firstTime max(_time)
as lastTime by EventCode Message ComputerName User | `security_content_ctime(firstTime)`
search: '`powershell` EventCode=4104 ScriptBlockText = "*[System.Security.Principal.WindowsIdentity]*" ScriptBlockText = "*GetCurrent()*"
| stats count min(_time) as firstTime max(_time) as lastTime by EventCode ScriptBlockText Computer user_id
| `security_content_ctime(firstTime)`
| `security_content_ctime(lastTime)`
| `getcurrent_user_with_powershell_script_block_filter`'
how_to_implement: To successfully implement this analytic, you will need to enable
PowerShell Script Block Logging on some or all endpoints. Additional setup here
@@ -30,7 +31,7 @@ tags:
- Source:Endpoint
- Stage:Discovery
dataset:
- https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1033/AD_discovery/windows-powershell.log
- https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1033/AD_discovery/windows-powershell-xml.log
impact: 30
kill_chain_phases:
- Reconnaissance
detections/endpoint/getlocaluser_with_powershell_script_block.yml
+7 -5
View File
@@ -1,7 +1,7 @@
name: GetLocalUser with PowerShell Script Block
id: 2e891cbe-0426-11ec-9c9c-acde48001122
version: 1
date: '2021-08-23'
version: 2
date: '2022-03-22'
author: Mauricio Velazco, Splunk
type: Hunting
datamodel: []
@@ -10,8 +10,10 @@ description: The following analytic utilizes PowerShell Script Block Logging (Ev
commandlet is used to return a list of all local users. Red Teams and adversaries
may leverage this commandlet to enumerate users for situational awareness and Active
Directory Discovery.
search: '`powershell` EventCode=4104 (Message = "*Get-LocalUser*") | stats count min(_time)
as firstTime max(_time) as lastTime by EventCode Message ComputerName User | `security_content_ctime(firstTime)`
search: '`powershell` EventCode=4104 (ScriptBlockText = "*Get-LocalUser*")
| stats count min(_time) as firstTime max(_time) as lastTime by EventCode ScriptBlockText Computer user_id
| `security_content_ctime(firstTime)`
| `security_content_ctime(lastTime)`
| `getlocaluser_with_powershell_script_block_filter`'
how_to_implement: To successfully implement this analytic, you will need to enable
PowerShell Script Block Logging on some or all endpoints. Additional setup here
@@ -28,7 +30,7 @@ tags:
- Source:Endpoint
- Stage:Discovery
dataset:
- https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1087.001/AD_discovery/windows-powershell.log
- https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1087.001/AD_discovery/windows-powershell-xml.log
impact: 30
kill_chain_phases:
- Reconnaissance
detections/endpoint/interactive_session_on_remote_endpoint_with_powershell.yml
+8 -6
View File
@@ -1,7 +1,7 @@
name: Interactive Session on Remote Endpoint with PowerShell
id: a4e8f3a4-48b2-11ec-bcfc-3e22fbd008af
version: 2
date: '2022-02-18'
version: 3
date: '2022-03-22'
author: Mauricio Velazco, Splunk
type: TTP
datamodel: []
@@ -10,9 +10,11 @@ description: The following analytic utilizes PowerShell Script Block Logging (Ev
an interactive session on a remote endpoint leveraging the WinRM protocol. Red Teams
and adversaries alike may abuse WinRM and `Enter-PSSession` for lateral movement
and remote code execution.
search: '`powershell` EventCode=4104 (Message="*Enter-PSSession*" AND Message="*-ComputerName*")
| stats count min(_time) as firstTime max(_time) as lastTime by EventCode Message
ComputerName User | `security_content_ctime(firstTime)` | `interactive_session_on_remote_endpoint_with_powershell_filter`'
search: '`powershell` EventCode=4104 (ScriptBlockText="*Enter-PSSession*" AND ScriptBlockText="*-ComputerName*")
| stats count min(_time) as firstTime max(_time) as lastTime by EventCode ScriptBlockText Computer user_id
| `security_content_ctime(firstTime)`
| `security_content_ctime(lastTime)`
| `interactive_session_on_remote_endpoint_with_powershell_filter`'
how_to_implement: To successfully implement this analytic, you will need to enable
PowerShell Script Block Logging on some or all endpoints. Additional setup instructions
can be found https://docs.splunk.com/Documentation/UBA/5.0.4.1/GetDataIn/AddPowerShell#Configure_module_logging_for_PowerShell.
@@ -30,7 +32,7 @@ tags:
- Source:Endpoint
- Stage:Lateral Movement
dataset:
- https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1021.006/lateral_movement_pssession/windows-powershell.log
- https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1021.006/lateral_movement_pssession/windows-powershell-xml.log
impact: 90
kill_chain_phases:
- Exploitation
detections/endpoint/kerberos_pre_authentication_flag_disabled_with_powershell.yml
+8 -6
View File
@@ -1,7 +1,7 @@
name: Kerberos Pre-Authentication Flag Disabled with PowerShell
id: 59b51620-94c9-11ec-b3d5-acde48001122
version: 1
date: '2022-02-23'
version: 2
date: '2022-03-22'
author: Mauricio Velazco, Splunk
type: TTP
datamodel: []
@@ -15,9 +15,11 @@ description: The following analytic utilizes PowerShell Script Block Logging (Ev
against the user's password offline leveraging the ASP REP Roasting technique. Red
Teams and adversaries alike who have obtained privileges in an Active Directory
network may use this technique as a backdoor or a way to escalate privileges.
search: ' `powershell` EventCode=4104 (Message = "*Set-ADAccountControl*" AND Message="*DoesNotRequirePreAuth:$true*")
| stats count min(_time) as firstTime max(_time) as lastTime by EventCode Message
ComputerName User | `security_content_ctime(firstTime)` | `kerberos_pre_authentication_flag_disabled_with_powershell_filter`'
search: '`powershell` EventCode=4104 (ScriptBlockText = "*Set-ADAccountControl*" AND ScriptBlockText="*DoesNotRequirePreAuth:$true*")
| stats count min(_time) as firstTime max(_time) as lastTime by EventCode ScriptBlockText Computer user_id
| `security_content_ctime(firstTime)`
| `security_content_ctime(lastTime)`
| `kerberos_pre_authentication_flag_disabled_with_powershell_filter`'
how_to_implement: To successfully implement this analytic, you will need to enable
PowerShell Script Block Logging on some or all endpoints. Additional setup here
https://docs.splunk.com/Documentation/UBA/5.0.4.1/GetDataIn/AddPowerShell#Configure_module_logging_for_PowerShell.
@@ -37,7 +39,7 @@ tags:
- Stage:Privilege Escalation
- Stage:Persistence
dataset:
- https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1558.004/powershell/windows-powershell.log
- https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1558.004/powershell/windows-powershell-xml.log
impact: 50
kill_chain_phases:
- Exploitation
detections/endpoint/linux_deletion_of_ssh_key.yml → detections/endpoint/linux_deletion_of_ssh_hash_conf.yml
+17 -17
View File
@@ -1,4 +1,4 @@
name: Linux deletion Of SSH Key
name: Linux deletion Of SSH Hash Conf
id: 73a56508-1cf5-4df7-b8d9-5737fbdc27d2
version: 1
date: '2022-04-12'
@@ -6,24 +6,24 @@ author: Teoderick Contreras, Splunk
type: Anomaly
datamodel:
- Endpoint
description: This analytic is to detect a deletion of ssh key in a linux machine.
attacker may delete or modify ssh key to impair some security features or act as defense evasion in compromised linux machine.
This Anomaly can be also a good indicator of a malware trying to wipe or delete several files in a compromised host as part of its destructive payload like what
acidrain malware does in linux or router machines. This detection can be a good pivot to check what process and user
description: This analytic is to detect a deletion of ssh key in a linux machine.
attacker may delete or modify ssh key to impair some security features or act as defense evasion in compromised linux machine.
This Anomaly can be also a good indicator of a malware trying to wipe or delete several files in a compromised host as part of its destructive payload like what
acidrain malware does in linux or router machines. This detection can be a good pivot to check what process and user
tries to delete this type of files which is not so common and need further investigation.
search: '| tstats `security_content_summariesonly` count min(_time) as firstTime max(_time) as lastTime FROM datamodel=Endpoint.Filesystem
where Filesystem.action=deleted Filesystem.file_path = "/etc/ssh/*"
search: '| tstats `security_content_summariesonly` count min(_time) as firstTime max(_time) as lastTime FROM datamodel=Endpoint.Filesystem
where Filesystem.action=deleted AND Filesystem.file_path IN ("/etc/ssh/*", "~/.ssh/*")
by _time span=1h Filesystem.file_name Filesystem.file_path Filesystem.dest Filesystem.process_guid Filesystem.action
| `drop_dm_object_name(Filesystem)`
|rename process_guid as proc_guid
| `drop_dm_object_name(Filesystem)`
|rename process_guid as proc_guid
|join proc_guid, _time [
| tstats `security_content_summariesonly` count FROM datamodel=Endpoint.Processes where Processes.parent_process_name != unknown
by _time span=1h Processes.process_id Processes.process_name Processes.process Processes.dest Processes.parent_process_name Processes.parent_process Processes.process_path Processes.process_guid
| `drop_dm_object_name(Processes)`
|rename process_guid as proc_guid
| fields _time dest user parent_process_name parent_process process_name process_path process proc_guid registry_path registry_value_name registry_value_data registry_key_name action]
| tstats `security_content_summariesonly` count FROM datamodel=Endpoint.Processes where Processes.parent_process_name != unknown
by _time span=1h Processes.process_id Processes.process_name Processes.process Processes.dest Processes.parent_process_name Processes.parent_process Processes.process_path Processes.process_guid
| `drop_dm_object_name(Processes)`
|rename process_guid as proc_guid
| fields _time dest user parent_process_name parent_process process_name process_path process proc_guid registry_path registry_value_name registry_value_data registry_key_name action]
| table process_name process proc_guid file_name file_path action _time parent_process_name parent_process process_path dest user
| `linux_deletion_of_ssh_key_filter`'
| `linux_deletion_of_ssh_hash_conf_filter`'
how_to_implement: To successfully implement this search, you need to be ingesting
logs with the process name, parent process, and command-line executions from your
endpoints. If you are using Sysmon, you can use the Add-on for Linux Sysmon from
@@ -31,7 +31,7 @@ how_to_implement: To successfully implement this search, you need to be ingestin
known_false_positives: Administrator or network operator can execute this command.
Please update the filter macros to remove false positives.
references:
- https://www.sentinelone.com/labs/acidrain-a-modem-wiper-rains-down-on-europe/
- https://www.sentinelone.com/labs/acidrain-a-modem-wiper-rains-down-on-europe/
tags:
analytic_story:
- Acidrain
@@ -83,4 +83,4 @@ tags:
- Processes.process_id
- Processes.parent_process_id
risk_score: 49
security_domain: endpoint
security_domain: endpoint
detections/endpoint/linux_iptables_firewall_modification.yml
+8 -8
View File
@@ -6,17 +6,17 @@ author: Teoderick Contreras, Splunk
type: Anomaly
datamodel:
- Endpoint
description: This analytic looks for suspicious commandline that modify the iptables firewall setting of a linux machine.
This technique was seen in cyclopsblink malware where it modifies the firewall setting of the compromised machine to allow
description: This analytic looks for suspicious commandline that modify the iptables firewall setting of a linux machine.
This technique was seen in cyclopsblink malware where it modifies the firewall setting of the compromised machine to allow
traffic to its tcp port that will be used to communicate with its C2 server.
search: '| tstats `security_content_summariesonly` count min(_time) as firstTime max(_time) as lastTime from datamodel=Endpoint.Processes
search: '| tstats `security_content_summariesonly` count min(_time) as firstTime max(_time) as lastTime from datamodel=Endpoint.Processes
where Processes.process = "*iptables *" AND Processes.process = "* --dport *" AND Processes.process = "* ACCEPT*" AND Processes.process = "*&amp;&gt;/dev/null*"
AND Processes.process = "* tcp *" AND NOT(Processes.parent_process_path IN("/bin/*", "/lib/*", "/usr/bin/*", "/sbin/*"))
by Processes.process_name Processes.process Processes.process_id Processes.parent_process_id Processes.process_guid Processes.dest
Processes.user Processes.parent_process_name Processes.parent_process_path Processes.process_path
by Processes.process_name Processes.process Processes.process_id Processes.parent_process_id Processes.process_guid Processes.dest
Processes.user Processes.parent_process_name Processes.parent_process_path Processes.process_path
| rex field=Processes.process "--dport (?<port>3269|636|989|994|995|8443)"
| stats values(Processes.process) as processes_exec values(port) as ports values(Processes.process_guid) as guids values(Processes.process_id) as pids dc(port) as port_count count by Processes.process_name Processes.parent_process_name Processes.parent_process_id Processes.dest Processes.user Processes.parent_process_path Processes.process_path firstTime lastTime | where port_count >=3
| `drop_dm_object_name(Processes)`
| `drop_dm_object_name(Processes)`
| `security_content_ctime(firstTime)`
| `security_content_ctime(lastTime)`
| `linux_iptables_firewall_modification_filter`'
@@ -24,14 +24,14 @@ how_to_implement: To successfully implement this search, you need to be ingestin
logs with the process name, parent process, and command-line executions from your
endpoints. If you are using Sysmon, you can use the Add-on for Linux Sysmon from
Splunkbase.
known_false_positives: administrator may do this commandline for auditing and testing purposes.
known_false_positives: administrator may do this commandline for auditing and testing purposes.
In this scenario filter is needed.
references:
- https://www.ncsc.gov.uk/files/Cyclops-Blink-Malware-Analysis-Report.pdf
- https://www.trendmicro.com/en_us/research/22/c/cyclops-blink-sets-sights-on-asus-routers--.html
tags:
analytic_story:
- Cyclops BLink
- CyclopsBLink
asset_type: Endpoint
cis20:
- CIS 3
detections/endpoint/linux_kworker_process_in_writable_process_path.yml
+10 -10
View File
@@ -6,18 +6,18 @@ author: Teoderick Contreras, Splunk
type: Hunting
datamodel:
- Endpoint
description: This analytic looks for suspicious process kworker commandline in a linux machine.
kworker process name or thread are common names of kernel threads in linux process.
This hunting detections can lead to investigate process contains process path in
writable directory in linux like /home/, /var/log and /tmp/. This technique was seen
in cyclopsblink malware to blend its core and other of its child process as normal
kworker on the compromised machine. This detection might be a good pivot to look for
description: This analytic looks for suspicious process kworker commandline in a linux machine.
kworker process name or thread are common names of kernel threads in linux process.
This hunting detections can lead to investigate process contains process path in
writable directory in linux like /home/, /var/log and /tmp/. This technique was seen
in cyclopsblink malware to blend its core and other of its child process as normal
kworker on the compromised machine. This detection might be a good pivot to look for
other IOC related to cyclopsblink malware or attacks.
search: '| tstats `security_content_summariesonly` count min(_time) as firstTime max(_time) as lastTime from datamodel=Endpoint.Processes
search: '| tstats `security_content_summariesonly` count min(_time) as firstTime max(_time) as lastTime from datamodel=Endpoint.Processes
where Processes.parent_process = "*[kworker/*" Processes.parent_process_path IN ("/home/*", "/tmp/*", "/var/log/*") Processes.process="*iptables*"
by Processes.parent_process_name Processes.parent_process Processes.process_name Processes.process Processes.process_id Processes.parent_process_id Processes.parent_process_path
Processes.process_guid Processes.dest Processes.user
| `drop_dm_object_name(Processes)`
Processes.process_guid Processes.dest Processes.user
| `drop_dm_object_name(Processes)`
| `security_content_ctime(firstTime)`
| `security_content_ctime(lastTime)`
| `linux_kworker_process_in_writable_process_path_filter`'
@@ -31,7 +31,7 @@ references:
- https://www.trendmicro.com/en_us/research/22/c/cyclops-blink-sets-sights-on-asus-routers--.html
tags:
analytic_story:
- Cyclops BLink
- CyclopsBLink
asset_type: Endpoint
cis20:
- CIS 3
detections/endpoint/powershell_execute_com_object.yml
+8 -6
View File
@@ -1,7 +1,7 @@
name: Powershell Execute COM Object
id: 65711630-f9bf-11eb-8d72-acde48001122
version: 1
date: '2021-08-10'
version: 2
date: '2022-03-22'
author: Teoderick Contreras, Splunk
type: TTP
datamodel:
@@ -11,10 +11,11 @@ description: This search is to detect a COM CLSID execution through powershell.
it has a feature to execute command using COM Object. This technique may use by
network operator at some cases but a good indicator if some application want to
gain privilege escalation or bypass uac.
search: '`powershell` EventCode=4104 Message = "*CreateInstance([type]::GetTypeFromCLSID*"
OR Message = "*CreateInstance([Type]::GetTypeFromProgID*"| stats count min(_time)
as firstTime max(_time) as lastTime by EventCode Message ComputerName User | `security_content_ctime(firstTime)`
| `security_content_ctime(lastTime)` | `powershell_execute_com_object_filter`'
search: '`powershell` EventCode=4104 ScriptBlockText = "*CreateInstance([type]::GetTypeFromCLSID*"
| stats count min(_time) as firstTime max(_time) as lastTime by EventCode ScriptBlockText Computer user_id
| `security_content_ctime(firstTime)`
| `security_content_ctime(lastTime)`
| `powershell_execute_com_object_filter`'
how_to_implement: To successfully implement this search, you need to be ingesting
logs with the process name, parent process, and command-line executions from your
endpoints. If you are using Sysmon, you must have at least version 6.0.4 of the
@@ -32,6 +33,7 @@ tags:
- Source:Endpoint
- Stage:Privilege Escalation
dataset:
- https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1546.015/pwh_com_object/windows-powershell-xml.log
- https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/malware/conti/conti_leak/windows-powershell.log
impact: 10
kill_chain_phases:
detections/endpoint/powershell_using_memory_as_backing_store.yml
+9 -7
View File
@@ -1,7 +1,7 @@
name: Powershell Using memory As Backing Store
id: c396a0c4-c9f2-11eb-b4f5-acde48001122
version: 1
date: '2021-06-10'
version: 2
date: '2022-03-22'
author: Teoderick Contreras, Splunk
type: TTP
datamodel: []
@@ -10,9 +10,10 @@ description: The following analytic identifies suspicious PowerShell script exec
PowerShell script will contain stream flate data and will be decompressed in memory
to run or drop the actual payload. During triage, review parallel processes within
the same timeframe. Review the full script block to identify other related artifacts.
search: '`powershell` EventCode=4104 Message = "*New-Object IO.MemoryStream*" | stats
count min(_time) as firstTime max(_time) as lastTime by EventCode Message ComputerName
User | `security_content_ctime(firstTime)` | `security_content_ctime(lastTime)`
search: '`powershell` EventCode=4104 ScriptBlockText = *New-Object* ScriptBlockText = *IO.MemoryStream*
| stats count min(_time) as firstTime max(_time) as lastTime by EventCode ScriptBlockText Computer user_id
| `security_content_ctime(firstTime)`
| `security_content_ctime(lastTime)`
| `powershell_using_memory_as_backing_store_filter`'
how_to_implement: To successfully implement this analytic, you will need to enable
PowerShell Script Block Logging on some or all endpoints. Additional setup here
@@ -34,14 +35,15 @@ tags:
- Source:Endpoint
- Stage:Defense Evasion
dataset:
- https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/honeypots/pwsh/windows-powershell.log
- https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/honeypots/pwsh/windows-powershell-xml.log
impact: 50
kill_chain_phases:
- Exploitation
message: A suspicious powershell script contains memorystream command in $Message$
as new object backstore with EventCode $EventCode$ in host $ComputerName$
mitre_attack_id:
- T1140
- T1059.001
- T1059
observable:
- name: ComputerName
type: Hostname
detections/endpoint/recon_avproduct_through_pwh_or_wmi.yml
+8 -6
View File
@@ -1,7 +1,7 @@
name: Recon AVProduct Through Pwh or WMI
id: 28077620-c9f6-11eb-8785-acde48001122
version: 1
date: '2021-06-10'
version: 2
date: '2022-03-22'
author: Teoderick Contreras, Splunk
type: TTP
datamodel: []
@@ -11,10 +11,11 @@ description: The following analytic identifies suspicious PowerShell script exec
adversary will map all running security applications or services. During triage,
review parallel processes within the same timeframe. Review the full script block
to identify other related artifacts.
search: '`powershell` EventCode=4104 (Message = "*SELECT*" OR Message = "*WMIC*")
AND (Message = "*AntiVirusProduct*" OR Message = "*AntiSpywareProduct*") | stats
count min(_time) as firstTime max(_time) as lastTime by EventCode Message ComputerName
User | `security_content_ctime(firstTime)` | `security_content_ctime(lastTime)`
search: '`powershell` EventCode=4104 (ScriptBlockText = "*SELECT*" OR ScriptBlockText = "*WMIC*")
AND (ScriptBlockText = "*AntiVirusProduct*" OR ScriptBlockText = "*AntiSpywareProduct*")
| stats count min(_time) as firstTime max(_time) as lastTime by EventCode ScriptBlockText Computer user_id
| `security_content_ctime(firstTime)`
| `security_content_ctime(lastTime)`
| `recon_avproduct_through_pwh_or_wmi_filter`'
how_to_implement: To successfully implement this analytic, you will need to enable
PowerShell Script Block Logging on some or all endpoints. Additional setup here
@@ -36,6 +37,7 @@ tags:
- Source:Endpoint
- Stage:Discovery
dataset:
- https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/t1592/pwh_av_recon/windows-powershell-xml.log
- https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/honeypots/pwsh/windows-powershell.log
impact: 70
kill_chain_phases:
detections/endpoint/remote_process_instantiation_via_dcom_and_powershell_script_block.yml
+6 -5
View File
@@ -1,7 +1,7 @@
name: Remote Process Instantiation via DCOM and PowerShell Script Block
id: fa1c3040-4680-11ec-a618-3e22fbd008af
version: 1
date: '2021-11-15'
version: 2
date: '2022-03-22'
author: Mauricio Velazco, Splunk
type: TTP
datamodel: []
@@ -10,9 +10,10 @@ description: The following analytic utilizes PowerShell Script Block Logging (Ev
on a remote endpoint by abusing the DCOM protocol. Specifically, this search looks
for the abuse of ShellExecute and ExecuteShellCommand. Red Teams and adversaries
alike may abuse DCOM for lateral movement and remote code execution.
search: '`powershell` EventCode=4104 (Message="*Document.Application.ShellExecute*"
OR Message="*Document.ActiveView.ExecuteShellCommand*") | stats count min(_time)
as firstTime max(_time) as lastTime by EventCode Message ComputerName User | `security_content_ctime(firstTime)`
search: '`powershell` EventCode=4104 (ScriptBlockText="*Document.Application.ShellExecute*" OR ScriptBlockText="*Document.ActiveView.ExecuteShellCommand*")
| stats count min(_time) as firstTime max(_time) as lastTime by EventCode ScriptBlockText Computer user_id
| `security_content_ctime(firstTime)`
| `security_content_ctime(lastTime)`
| `remote_process_instantiation_via_dcom_and_powershell_script_block_filter`'
how_to_implement: To successfully implement this analytic, you will need to enable
PowerShell Script Block Logging on some or all endpoints. Additional setup instructions
detections/endpoint/remote_process_instantiation_via_winrm_and_powershell_script_block.yml
+8 -6
View File
@@ -1,7 +1,7 @@
name: Remote Process Instantiation via WinRM and PowerShell Script Block
id: 7d4c618e-4716-11ec-951c-3e22fbd008af
version: 1
date: '2021-11-16'
version: 2
date: '2022-03-22'
author: Mauricio Velazco, Splunk
type: TTP
datamodel: []
@@ -10,9 +10,11 @@ description: The following analytic utilizes PowerShell Script Block Logging (Ev
on a remote endpoint by abusing the WinRM protocol. Specifically, this search looks
for the abuse of the `Invoke-Command` commandlet. Red Teams and adversaries alike
may abuse WinRM for lateral movement and remote code execution.
search: '`powershell` EventCode=4104 (Message="*Invoke-Command*" AND Message="*-ComputerName*")
| stats count min(_time) as firstTime max(_time) as lastTime by EventCode Message
ComputerName User | `security_content_ctime(firstTime)` | `remote_process_instantiation_via_winrm_and_powershell_script_block_filter`'
search: '`powershell` EventCode=4104 (ScriptBlockText="*Invoke-Command*" AND ScriptBlockText="*-ComputerName*")
| stats count min(_time) as firstTime max(_time) as lastTime by EventCode ScriptBlockText Computer user_id
| `security_content_ctime(firstTime)`
| `security_content_ctime(lastTime)`
| `remote_process_instantiation_via_winrm_and_powershell_script_block_filter`'
how_to_implement: To successfully implement this analytic, you will need to enable
PowerShell Script Block Logging on some or all endpoints. Additional setup instructions
can be found https://docs.splunk.com/Documentation/UBA/5.0.4.1/GetDataIn/AddPowerShell#Configure_module_logging_for_PowerShell.
@@ -31,7 +33,7 @@ tags:
- Source:Endpoint
- Stage:Lateral Movement
dataset:
- https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1021.006/lateral_movement_psh/windows-powershell.log
- https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1021.006/lateral_movement_psh/windows-powershell-xml.log
impact: 90
kill_chain_phases:
- Exploitation
detections/endpoint/unloading_amsi_via_reflection.yml
+7 -4
View File
@@ -19,9 +19,10 @@ description: 'The following analytic utilizes PowerShell Script Block Logging (E
During triage, review parallel processes using an EDR product or 4688 events. It
will be important to understand the timeline of events around this activity. Review
the entire logged PowerShell script block.'
search: '`powershell` EventCode=4104 Message=*system.management.automation.amsi* |
stats count min(_time) as firstTime max(_time) as lastTime by OpCode ComputerName
User EventCode Message | `security_content_ctime(firstTime)` | `security_content_ctime(lastTime)`
search: '`powershell` EventCode=4104 ScriptBlockText = *system.management.automation.amsi*
| stats count min(_time) as firstTime max(_time) as lastTime by EventCode ScriptBlockText Computer user_id
| `security_content_ctime(firstTime)`
| `security_content_ctime(lastTime)`
| `unloading_amsi_via_reflection_filter`'
how_to_implement: To successfully implement this analytic, you will need to enable
PowerShell Script Block Logging on some or all endpoints. Additional setup here
@@ -43,13 +44,15 @@ tags:
- Stage:Execution
- Stage:Defense Evasion
dataset:
- https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1059.001/powershell_script_block_logging/windows-powershell.log
- https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1059.001/powershell_script_block_logging/windows-powershell-xml.log
impact: 70
kill_chain_phases:
- Exploitation
message: Possible AMSI Unloading via Reflection using PowerShell on $ComputerName$
mitre_attack_id:
- T1562
- T1059.001
- T1059
observable:
- name: ComputerName
type: Endpoint
detections/endpoint/user_discovery_with_env_vars_powershell_script_block.yml
+8 -6
View File
@@ -1,7 +1,7 @@
name: User Discovery With Env Vars PowerShell Script Block
id: 77f41d9e-b8be-47e3-ab35-5776f5ec1d20
version: 1
date: '2021-09-13'
version: 2
date: '2022-03-22'
author: Mauricio Velazco, Splunk
type: Hunting
datamodel: []
@@ -10,9 +10,11 @@ description: The following analytic utilizes PowerShell Script Block Logging (Ev
logged user. Red Teams and adversaries may leverage this method to identify the
logged user on a compromised endpoint for situational awareness and Active Directory
Discovery.
search: '`powershell` EventCode=4104 (Message = "*$env:UserName*" OR Message = "*[System.Environment]::UserName*")
| stats count min(_time) as firstTime max(_time) as lastTime by EventCode Message
ComputerName User | `security_content_ctime(firstTime)` | `user_discovery_with_env_vars_powershell_script_block_filter`'
search: '`powershell` EventCode=4104 (ScriptBlockText = "*$env:UserName*" OR ScriptBlockText = "*[System.Environment]::UserName*")
| stats count min(_time) as firstTime max(_time) as lastTime by EventCode ScriptBlockText Computer user_id
| `security_content_ctime(firstTime)`
| `security_content_ctime(lastTime)`
| `user_discovery_with_env_vars_powershell_script_block_filter`'
how_to_implement: To successfully implement this analytic, you will need to enable
PowerShell Script Block Logging on some or all endpoints. Additional setup here
https://docs.splunk.com/Documentation/UBA/5.0.4.1/GetDataIn/AddPowerShell#Configure_module_logging_for_PowerShell.
@@ -29,7 +31,7 @@ tags:
- Stage:Discovery
- Stage:Recon
dataset:
- https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1033/AD_discovery/windows-powershell.log
- https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1033/AD_discovery/windows-powershell-xml.log
impact: 30
kill_chain_phases:
- Reconnaissance
detections/experimental/endpoint/linux_stdout_redirection_to_dev_null_file.yml
+8 -8
View File
@@ -6,16 +6,16 @@ author: Teoderick Contreras, Splunk
type: Anomaly
datamodel:
- Endpoint
description: This analytic looks for suspicious commandline that redirect the stdout or possible stderror to dev/null file.
This technique was seen in cyclopsblink malware where it redirect the possible output or error while modify the iptables firewall setting of the
description: This analytic looks for suspicious commandline that redirect the stdout or possible stderror to dev/null file.
This technique was seen in cyclopsblink malware where it redirect the possible output or error while modify the iptables firewall setting of the
compromised machine to hide its action from the user. This Anomaly detection is a good pivot to look further why process or user use this un common approach.
search: '| tstats `security_content_summariesonly` count min(_time) as firstTime max(_time) as lastTime from datamodel=Endpoint.Processes
where Processes.process = "*&amp;&gt;/dev/null*"
by Processes.process_name Processes.process Processes.process_id Processes.parent_process_id
search: '| tstats `security_content_summariesonly` count min(_time) as firstTime max(_time) as lastTime from datamodel=Endpoint.Processes
where Processes.process = "*&amp;&gt;/dev/null*"
by Processes.process_name Processes.process Processes.process_id Processes.parent_process_id
Processes.process_guid Processes.dest Processes.user Processes.parent_process_name
| `drop_dm_object_name(Processes)`
| `drop_dm_object_name(Processes)`
| `security_content_ctime(firstTime)`
| `security_content_ctime(lastTime)`
| `security_content_ctime(lastTime)`
| `linux_stdout_redirection_to_dev_null_file_filter`'
how_to_implement: To successfully implement this search, you need to be ingesting
logs with the process name, parent process, and command-line executions from your
@@ -27,7 +27,7 @@ references:
- https://www.trendmicro.com/en_us/research/22/c/cyclops-blink-sets-sights-on-asus-routers--.html
tags:
analytic_story:
- Cyclops BLink
- CyclopsBLink
- Industroyer2
asset_type: Endpoint
cis20:
detections/network/f5_big_ip_icontrol_rest_vulnerability_cve_2022_1388.yml
+68
View File
@@ -0,0 +1,68 @@
name: F5 BIG-IP iControl REST Vulnerability CVE-2022-1388
id: bb1c2c30-107a-4e56-a4b9-1f7022867bfe
version: 1
date: '2022-05-10'
author: Michael Haag, Splunk
type: TTP
datamodel:
- Web
description: The following analytic identifies a recent unauthenticated remote code execution vulnerablity against the F5 BIG-IP iControl REST API. The analytic identifies the URI path found in the POCs and the HTTP Method of POST.
In addition, the request header will have the commands that may be executed in fields utilcmdargs and the auth field of X-F5-Auth-Token, which may have a random base64 encoded value.
search: '| tstats count from datamodel=Web where
Web.url="*/mgmt/tm/util/bash*" Web.http_method="POST" by Web.http_user_agent Web.http_method, Web.url,Web.url_length
Web.src, Web.dest | `drop_dm_object_name("Web")`
| `security_content_ctime(firstTime)`
| `security_content_ctime(lastTime)`
| `f5_big_ip_icontrol_rest_vulnerability_cve_2022_1388_filter`'
how_to_implement: To successfully implement this search, you need to be ingesting
web or proxy logs, or ensure it is being filled by a proxy like device, into the
Web Datamodel. For additional filtering, allow list private IP space or restrict
by known good.
known_false_positives: False positives may be present if the activity is blocked or was not successful. Filter known vulnerablity scanners. Filter as needed.
references:
- https://github.com/dk4trin/templates-nuclei/blob/main/CVE-2022-1388.yaml
- https://www.randori.com/blog/vulnerability-analysis-cve-2022-1388/
- https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-1388
- https://twitter.com/da_667/status/1523770267327250438?s=20&t=-JnB_aNWuJFsmcOmxGUWLQ
- https://github.com/horizon3ai/CVE-2022-1388/blob/main/CVE-2022-1388.py
tags:
analytic_story:
- F5 BIG-IP Vulnerability CVE-2022-1388
asset_type: Web Server
cis20:
- CIS 3
- CIS 5
- CIS 16
confidence: 70
context:
- Scope:Network
dataset:
- https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1190/f5/f5.log
impact: 100
kill_chain_phases:
- Exploitation
message: An attempt to exploit CVE-2022-1388 against an F5 appliance $dest$ has occurred.
mitre_attack_id:
- T1190
cve:
- CVE-2022-1388
nist:
- DE.CM
observable:
- name: dest
type: Hostname
role:
- Victim
product:
- Splunk Enterprise
- Splunk Enterprise Security
- Splunk Cloud
required_fields:
- Web.http_method
- Web.url
- Web.url_length
- Web.src
- Web.dest
- Web.http_user_agent
risk_score: 70
security_domain: network
dist/escu/bin/dnstwist.py
Vendored
-400
View File
@@ -1,400 +0,0 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-
#
#
#
# Licensed under the Apache License, Version 2.0 (the "License"): you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
#
# Code modified from DNSTwist (https://github.com/elceef/dnstwist)
# Thanks elceef!
#
# Changes made:
# Just kept the DomainFuzz class and passing the domain to the fuzzer. Then added
# the Splunk specific code around it
#
from __future__ import absolute_import, division, print_function, unicode_literals
import sys
import re
import csv
import time
import os
from splunklib.searchcommands import dispatch, GeneratingCommand, \
Configuration, Option, Boolean
from splunk.clilib.bundle_paths import make_splunkhome_path
class DomainFuzz(object):
def __init__(self, domain):
self.domain, self.tld = self.__domain_tld(domain)
self.domains = []
self.qwerty = {
'1': '2q', '2': '3wq1', '3': '4ew2', '4': '5re3',
'5': '6tr4', '6': '7yt5', '7': '8uy6', '8': '9iu7',
'9': '0oi8', '0': 'po9', 'q': '12wa', 'w': '3esaq2',
'e': '4rdsw3', 'r': '5tfde4', 't': '6ygfr5', 'y': '7uhgt6',
'u': '8ijhy7', 'i': '9okju8', 'o': '0plki9', 'p': 'lo0',
'a': 'qwsz', 's': 'edxzaw', 'd': 'rfcxse', 'f': 'tgvcdr',
'g': 'yhbvft', 'h': 'ujnbgy', 'j': 'ikmnhu', 'k': 'olmji',
'l': 'kop', 'z': 'asx', 'x': 'zsdc', 'c': 'xdfv',
'v': 'cfgb', 'b': 'vghn', 'n': 'bhjm', 'm': 'njk'
}
self.qwertz = {
'1': '2q', '2': '3wq1', '3': '4ew2', '4': '5re3',
'5': '6tr4', '6': '7zt5', '7': '8uz6', '8': '9iu7',
'9': '0oi8', '0': 'po9', 'q': '12wa', 'w': '3esaq2',
'e': '4rdsw3', 'r': '5tfde4', 't': '6zgfr5',
'z': '7uhgt6', 'u': '8ijhz7', 'i': '9okju8',
'o': '0plki9', 'p': 'lo0', 'a': 'qwsy', 's': 'edxyaw',
'd': 'rfcxse', 'f': 'tgvcdr', 'g': 'zhbvft',
'h': 'ujnbgz', 'j': 'ikmnhu', 'k': 'olmji', 'l': 'kop',
'y': 'asx', 'x': 'ysdc', 'c': 'xdfv', 'v': 'cfgb',
'b': 'vghn', 'n': 'bhjm', 'm': 'njk'
}
self.azerty = {
'1': '2a', '2': '3za1', '3': '4ez2', '4': '5re3',
'5': '6tr4', '6': '7yt5', '7': '8uy6', '8': '9iu7',
'9': '0oi8', '0': 'po9', 'a': '2zq1', 'z': '3esqa2',
'e': '4rdsz3', 'r': '5tfde4', 't': '6ygfr5',
'y': '7uhgt6', 'u': '8ijhy7', 'i': '9okju8',
'o': '0plki9', 'p': 'lo0m', 'q': 'zswa', 's': 'edxwqz',
'd': 'rfcxse', 'f': 'tgvcdr', 'g': 'yhbvft',
'h': 'ujnbgy', 'j': 'iknhu', 'k': 'olji', 'l': 'kopm',
'm': 'lp', 'w': 'sxq', 'x': 'zsdc', 'c': 'xdfv',
'v': 'cfgb', 'b': 'vghn', 'n': 'bhj'
}
self.keyboards = [self.qwerty, self.qwertz, self.azerty]
def __domain_tld(self, domain):
domain = domain.rsplit('.', 2)
if len(domain) == 2:
return domain[0], domain[1]
return domain[0] + '.' + domain[1], domain[2]
def __validate_domain(self, domain):
if len(domain) == len(domain.encode('idna')) and domain != domain.encode('idna'):
return False
allowed = re.compile(b'(?=^.{4,253}$)(^((?!-)[a-zA-Z0-9-]{1,63}(?<!-)\\.)+[a-zA-Z]{2,63}\\.?$)', re.IGNORECASE)
return allowed.match(domain.encode('idna'))
def __filter_domains(self):
seen = set()
filtered = []
for d in self.domains:
# if not self.__validate_domain(d['domain-name']):
# p_err("debug: invalid domain %s\n" % d['domain-name'])
try:
if self.__validate_domain(d['domain-name']) and d['domain-name'] not in seen:
seen.add(d['domain-name'])
filtered.append(d)
except ValueError:
continue
self.domains = filtered
def __bitsquatting(self):
result = []
masks = [1, 2, 4, 8, 16, 32, 64, 128]
for i in range(0, len(self.domain)):
c = self.domain[i]
for j in range(0, len(masks)):
b = chr(ord(c) ^ masks[j])
o = ord(b)
if (o >= 48 and o <= 57) or (o >= 97 and o <= 122) or o == 45:
result.append(self.domain[:i] + b + self.domain[i+1:])
return result
def __homoglyph(self):
glyphs = {
'a': [u'à', u'á', u'â', u'ã', u'ä', u'å', u'ɑ', u'а', u'ạ', u'ǎ', u'ă', u'ȧ', u'ӓ'],
'b': ['d', 'lb', 'ib', u'ʙ', u'Ь', u'b̔', u'ɓ', u'Б'],
'c': [u'ϲ', u'с', u'ƈ', u'ċ', u'ć', u'ç'],
'd': ['b', 'cl', 'dl', 'di', u'ԁ', u'ժ', u'ɗ', u'đ'],
'e': [u'é', u'ê', u'ë', u'ē', u'ĕ', u'ě', u'ė', u'е', u'ẹ', u'ę', u'є', u'ϵ', u'ҽ'],
'f': [u'Ϝ', u'ƒ', u'Ғ'],
'g': ['q', u'ɢ', u'ɡ', u'Ԍ', u'Ԍ', u'ġ', u'ğ', u'ց', u'ǵ', u'ģ'],
'h': ['lh', 'ih', u'һ', u'հ', u'Ꮒ', u'н'],
'i': ['1', 'l', u'Ꭵ', u'í', u'ï', u'ı', u'ɩ', u'ι', u'ꙇ', u'ǐ', u'ĭ'],
'j': [u'ј', u'ʝ', u'ϳ', u'ɉ'],
'k': ['lk', 'ik', 'lc', u'κ', u'ⲕ', u'κ'],
'l': ['1', 'i', u'ɫ', u'ł'],
'm': ['n', 'nn', 'rn', 'rr', u'ṃ', u'ᴍ', u'м', u'ɱ'],
'n': ['m', 'r', u'ń'],
'o': ['0', u'Ο', u'ο', u'О', u'о', u'Օ', u'ȯ', u'ọ', u'ỏ', u'ơ', u'ó', u'ö', u'ӧ'],
'p': [u'ρ', u'р', u'ƿ', u'Ϸ', u'Þ'],
'q': ['g', u'զ', u'ԛ', u'գ', u'ʠ'],
'r': [u'ʀ', u'Г', u'ᴦ', u'ɼ', u'ɽ'],
's': [u'Ⴝ', u'Ꮪ', u'ʂ', u'ś', u'ѕ'],
't': [u'τ', u'т', u'ţ'],
'u': [u'μ', u'υ', u'Ս', u'ս', u'ц', u'ᴜ', u'ǔ', u'ŭ'],
'v': [u'ѵ', u'ν', u'v̇'],
'w': ['vv', u'ѡ', u'ա', u'ԝ'],
'x': [u'х', u'ҳ', u'ẋ'],
'y': [u'ʏ', u'γ', u'у', u'Ү', u'ý'],
'z': [u'ʐ', u'ż', u'ź', u'ʐ', u'ᴢ']
}
result = []
for ws in range(0, len(self.domain)):
for i in range(0, (len(self.domain)-ws)+1):
win = self.domain[i:i+ws]
j = 0
while j < ws:
c = win[j]
if c in glyphs:
win_copy = win
for g in glyphs[c]:
win = win.replace(c, g)
result.append(self.domain[:i] + win + self.domain[i+ws:])
win = win_copy
j += 1
return list(set(result))
def __hyphenation(self):
result = []
for i in range(1, len(self.domain)):
result.append(self.domain[:i] + '-' + self.domain[i:])
return result
def __insertion(self):
result = []
for i in range(1, len(self.domain)-1):
for keys in self.keyboards:
if self.domain[i] in keys:
for c in keys[self.domain[i]]:
result.append(self.domain[:i] + c + self.domain[i] + self.domain[i+1:])
result.append(self.domain[:i] + self.domain[i] + c + self.domain[i+1:])
return list(set(result))
def __omission(self):
result = []
for i in range(0, len(self.domain)):
result.append(self.domain[:i] + self.domain[i+1:])
n = re.sub(r'(.)\1+', r'\1', self.domain)
if n not in result and n != self.domain:
result.append(n)
return list(set(result))
def __repetition(self):
result = []
for i in range(0, len(self.domain)):
if self.domain[i].isalpha():
result.append(self.domain[:i] + self.domain[i] + self.domain[i] + self.domain[i+1:])
return list(set(result))
def __replacement(self):
result = []
for i in range(0, len(self.domain)):
for keys in self.keyboards:
if self.domain[i] in keys:
for c in keys[self.domain[i]]:
result.append(self.domain[:i] + c + self.domain[i+1:])
return list(set(result))
def __subdomain(self):
result = []
for i in range(1, len(self.domain)):
if self.domain[i] not in ['-', '.'] and self.domain[i-1] not in ['-', '.']:
result.append(self.domain[:i] + '.' + self.domain[i:])
return result
def __transposition(self):
result = []
for i in range(0, len(self.domain)-1):
if self.domain[i+1] != self.domain[i]:
result.append(self.domain[:i] + self.domain[i+1] + self.domain[i] + self.domain[i+2:])
return result
def __vowel_swap(self):
vowels = 'aeiou'
result = []
for i in range(0, len(self.domain)):
for vowel in vowels:
if self.domain[i] in vowels:
result.append(self.domain[:i] + vowel + self.domain[i+1:])
return list(set(result))
def __addition(self):
result = []
for i in range(97, 123):
result.append(self.domain + chr(i))
return result
def generate(self):
self.domains.append({'fuzzer': 'Original*', 'domain-name': self.domain + '.' + self.tld})
for domain in self.__addition():
self.domains.append({'fuzzer': 'Addition', 'domain-name': domain + '.' + self.tld})
for domain in self.__bitsquatting():
self.domains.append({'fuzzer': 'Bitsquatting', 'domain-name': domain + '.' + self.tld})
for domain in self.__homoglyph():
self.domains.append({'fuzzer': 'Homoglyph', 'domain-name': domain + '.' + self.tld})
for domain in self.__hyphenation():
self.domains.append({'fuzzer': 'Hyphenation', 'domain-name': domain + '.' + self.tld})
for domain in self.__insertion():
self.domains.append({'fuzzer': 'Insertion', 'domain-name': domain + '.' + self.tld})
for domain in self.__omission():
self.domains.append({'fuzzer': 'Omission', 'domain-name': domain + '.' + self.tld})
for domain in self.__repetition():
self.domains.append({'fuzzer': 'Repetition', 'domain-name': domain + '.' + self.tld})
for domain in self.__replacement():
self.domains.append({'fuzzer': 'Replacement', 'domain-name': domain + '.' + self.tld})
for domain in self.__subdomain():
self.domains.append({'fuzzer': 'Subdomain', 'domain-name': domain + '.' + self.tld})
for domain in self.__transposition():
self.domains.append({'fuzzer': 'Transposition', 'domain-name': domain + '.' + self.tld})
for domain in self.__vowel_swap():
self.domains.append({'fuzzer': 'Vowel-swap', 'domain-name': domain + '.' + self.tld})
if not self.domain.startswith('www.'):
self.domains.append({'fuzzer': 'Various', 'domain-name': 'ww' + self.domain + '.' + self.tld})
self.domains.append({'fuzzer': 'Various', 'domain-name': 'www' + self.domain + '.' + self.tld})
self.domains.append({'fuzzer': 'Various', 'domain-name': 'www-' + self.domain + '.' + self.tld})
if '.' in self.tld:
self.domains.append({'fuzzer': 'Various', 'domain-name': self.domain + '.' + self.tld.split('.')[-1]})
self.domains.append({'fuzzer': 'Various', 'domain-name': self.domain + self.tld})
if '.' not in self.tld:
self.domains.append({'fuzzer': 'Various', 'domain-name': self.domain + self.tld + '.' + self.tld})
if self.tld != 'com' and '.' not in self.tld:
self.domains.append({'fuzzer': 'Various', 'domain-name': self.domain + '-' + self.tld + '.com'})
#self.__filter_domains()
@Configuration(distributed=True)
class DnsTwistCommand(GeneratingCommand):
domainlist_file_name = Option(doc='''
**Syntax:** **domainlist=***<path>*
**Description:** CSV file from which repeated random samples will be drawn
''', name='domainlist', require=False)
populate_from_cim = Option(doc='''
**Syntax: populate_cim=<bool>
**Description:** When `true`, populates Splunk_SA_CIM lookups cim_corporate_email_domains.csv
and cim_corporate_web_domains.csv with dnstwisted domains. Defaults to `false`.
''', name='populate_from_cim', default=False, validate=Boolean())
domain = Option(doc='''
**Syntax:** **domain=***<domain name>*
**Description:** Domain to DNS generated twisted entries for.
''', name='domain', require=False, default='')
def generate(self):
event_count = 0
csv_file_names = []
if self.populate_from_cim:
csv_file_names.append(make_splunkhome_path([
'etc',
'apps',
'Splunk_SA_CIM',
'lookups',
'cim_corporate_email_domains.csv']))
csv_file_names.append(make_splunkhome_path([
'etc',
'apps',
'Splunk_SA_CIM',
'lookups',
'cim_corporate_web_domains.csv']))
# Make sure we just get the base file name from file. In case there was some directory traversal going on.
if self.domainlist_file_name:
sanitized_file_name = os.path.basename(self.domainlist_file_name)
lookup_path = make_splunkhome_path(['etc', 'apps', 'DA-ESS-ContentUpdate', 'lookups', sanitized_file_name])
# Make sure there really isn't any directory traversal going on.
valid_path = True
if "../" in lookup_path:
valid_path = False
# Make sure the path that is created by adding the file name to the path is the same as the
# absolute path
if lookup_path != os.path.abspath(lookup_path):
valid_path = False
if valid_path:
csv_file_names.append(lookup_path)
domains_to_twist = []
for csv_file_name in csv_file_names:
if os.path.exists(csv_file_name):
# this is nasty but works .. please forgive me
if sys.version_info >= (3, 0):
csv_file = open(csv_file_name, "r", newline='')
else:
csv_file = open(csv_file_name, "r")
for input_domain in csv.DictReader(csv_file):
if input_domain['domain'] not in domains_to_twist:
domains_to_twist.append(input_domain['domain'])
# if a single domain is passed lets just calculate that
if self.domain != '':
domains_to_twist = []
domains_to_twist.append(self.domain)
for domain_to_twist in domains_to_twist:
domain_to_twist = domain_to_twist.lstrip('*')
dfuzz = DomainFuzz(domain_to_twist)
dfuzz.generate()
domains = dfuzz.domains
for domain in domains:
# We don't want to keep the original domain
if domain['domain-name'] in domain_to_twist:
continue
event_count += 1
yield {
'_time': time.time(),
'event_no': event_count,
'_raw': domain['domain-name'],
'domain': '*'+domain['domain-name']+'*',
'original_domain': domain_to_twist
}
def __init__(self):
super(DnsTwistCommand, self).__init__()
dispatch(DnsTwistCommand, sys.argv, sys.stdin, sys.stdout, __name__)
dist/escu/bin/escu_contextualize.py
Vendored
-1
View File
@@ -1 +0,0 @@
# dropped AR action support due to python 3 dependency, we leverage playbooks in stories as an alternative.
dist/escu/bin/escu_investigate.py
Vendored
-1
View File
@@ -1 +0,0 @@
# dropped AR action support due to python 3 dependency, we leverage playbooks in stories as an alternative.
dist/escu/bin/runstory.py
Vendored
-1
View File
@@ -1 +0,0 @@
# runstory was deprecated, its functionality was moved to: https://github.com/splunk/analytic_story_execution
dist/escu/bin/splunklib/__init__.py
Vendored
-20
View File
@@ -1,20 +0,0 @@
# Copyright 2011-2015 Splunk, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"): you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
"""Python library for Splunk."""
from __future__ import absolute_import
from splunklib.six.moves import map
__version_info__ = (1, 6, 16)
__version__ = ".".join(map(str, __version_info__))
dist/escu/bin/splunklib/binding.py
Vendored
-1415
View File
File diff suppressed because it is too large Load Diff
dist/escu/bin/splunklib/client.py
Vendored
-3737
View File
File diff suppressed because it is too large Load Diff
dist/escu/bin/splunklib/data.py
Vendored
-266
View File
@@ -1,266 +0,0 @@
# Copyright 2011-2015 Splunk, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"): you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
"""The **splunklib.data** module reads the responses from splunkd in Atom Feed
format, which is the format used by most of the REST API.
"""
from __future__ import absolute_import
import sys
from xml.etree.ElementTree import XML
from splunklib import six
__all__ = ["load"]
# LNAME refers to element names without namespaces; XNAME is the same
# name, but with an XML namespace.
LNAME_DICT = "dict"
LNAME_ITEM = "item"
LNAME_KEY = "key"
LNAME_LIST = "list"
XNAMEF_REST = "{http://dev.splunk.com/ns/rest}%s"
XNAME_DICT = XNAMEF_REST % LNAME_DICT
XNAME_ITEM = XNAMEF_REST % LNAME_ITEM
XNAME_KEY = XNAMEF_REST % LNAME_KEY
XNAME_LIST = XNAMEF_REST % LNAME_LIST
# Some responses don't use namespaces (eg: search/parse) so we look for
# both the extended and local versions of the following names.
def isdict(name):
return name == XNAME_DICT or name == LNAME_DICT
def isitem(name):
return name == XNAME_ITEM or name == LNAME_ITEM
def iskey(name):
return name == XNAME_KEY or name == LNAME_KEY
def islist(name):
return name == XNAME_LIST or name == LNAME_LIST
def hasattrs(element):
return len(element.attrib) > 0
def localname(xname):
rcurly = xname.find('}')
return xname if rcurly == -1 else xname[rcurly+1:]
def load(text, match=None):
"""This function reads a string that contains the XML of an Atom Feed, then
returns the
data in a native Python structure (a ``dict`` or ``list``). If you also
provide a tag name or path to match, only the matching sub-elements are
loaded.
:param text: The XML text to load.
:type text: ``string``
:param match: A tag name or path to match (optional).
:type match: ``string``
"""
if text is None: return None
text = text.strip()
if len(text) == 0: return None
nametable = {
'namespaces': [],
'names': {}
}
# Convert to unicode encoding in only python 2 for xml parser
if(sys.version_info < (3, 0, 0) and isinstance(text, unicode)):
text = text.encode('utf-8')
root = XML(text)
items = [root] if match is None else root.findall(match)
count = len(items)
if count == 0:
return None
elif count == 1:
return load_root(items[0], nametable)
else:
return [load_root(item, nametable) for item in items]
# Load the attributes of the given element.
def load_attrs(element):
if not hasattrs(element): return None
attrs = record()
for key, value in six.iteritems(element.attrib):
attrs[key] = value
return attrs
# Parse a <dict> element and return a Python dict
def load_dict(element, nametable = None):
value = record()
children = list(element)
for child in children:
assert iskey(child.tag)
name = child.attrib["name"]
value[name] = load_value(child, nametable)
return value
# Loads the given elements attrs & value into single merged dict.
def load_elem(element, nametable=None):
name = localname(element.tag)
attrs = load_attrs(element)
value = load_value(element, nametable)
if attrs is None: return name, value
if value is None: return name, attrs
# If value is simple, merge into attrs dict using special key
if isinstance(value, six.string_types):
attrs["$text"] = value
return name, attrs
# Both attrs & value are complex, so merge the two dicts, resolving collisions.
collision_keys = []
for key, val in six.iteritems(attrs):
if key in value and key in collision_keys:
value[key].append(val)
elif key in value and key not in collision_keys:
value[key] = [value[key], val]
collision_keys.append(key)
else:
value[key] = val
return name, value
# Parse a <list> element and return a Python list
def load_list(element, nametable=None):
assert islist(element.tag)
value = []
children = list(element)
for child in children:
assert isitem(child.tag)
value.append(load_value(child, nametable))
return value
# Load the given root element.
def load_root(element, nametable=None):
tag = element.tag
if isdict(tag): return load_dict(element, nametable)
if islist(tag): return load_list(element, nametable)
k, v = load_elem(element, nametable)
return Record.fromkv(k, v)
# Load the children of the given element.
def load_value(element, nametable=None):
children = list(element)
count = len(children)
# No children, assume a simple text value
if count == 0:
text = element.text
if text is None:
return None
text = text.strip()
if len(text) == 0:
return None
return text
# Look for the special case of a single well-known structure
if count == 1:
child = children[0]
tag = child.tag
if isdict(tag): return load_dict(child, nametable)
if islist(tag): return load_list(child, nametable)
value = record()
for child in children:
name, item = load_elem(child, nametable)
# If we have seen this name before, promote the value to a list
if name in value:
current = value[name]
if not isinstance(current, list):
value[name] = [current]
value[name].append(item)
else:
value[name] = item
return value
# A generic utility that enables "dot" access to dicts
class Record(dict):
"""This generic utility class enables dot access to members of a Python
dictionary.
Any key that is also a valid Python identifier can be retrieved as a field.
So, for an instance of ``Record`` called ``r``, ``r.key`` is equivalent to
``r['key']``. A key such as ``invalid-key`` or ``invalid.key`` cannot be
retrieved as a field, because ``-`` and ``.`` are not allowed in
identifiers.
Keys of the form ``a.b.c`` are very natural to write in Python as fields. If
a group of keys shares a prefix ending in ``.``, you can retrieve keys as a
nested dictionary by calling only the prefix. For example, if ``r`` contains
keys ``'foo'``, ``'bar.baz'``, and ``'bar.qux'``, ``r.bar`` returns a record
with the keys ``baz`` and ``qux``. If a key contains multiple ``.``, each
one is placed into a nested dictionary, so you can write ``r.bar.qux`` or
``r['bar.qux']`` interchangeably.
"""
sep = '.'
def __call__(self, *args):
if len(args) == 0: return self
return Record((key, self[key]) for key in args)
def __getattr__(self, name):
try:
return self[name]
except KeyError:
raise AttributeError(name)
def __delattr__(self, name):
del self[name]
def __setattr__(self, name, value):
self[name] = value
@staticmethod
def fromkv(k, v):
result = record()
result[k] = v
return result
def __getitem__(self, key):
if key in self:
return dict.__getitem__(self, key)
key += self.sep
result = record()
for k,v in six.iteritems(self):
if not k.startswith(key):
continue
suffix = k[len(key):]
if '.' in suffix:
ks = suffix.split(self.sep)
z = result
for x in ks[:-1]:
if x not in z:
z[x] = record()
z = z[x]
z[ks[-1]] = v
else:
result[suffix] = v
if len(result) == 0:
raise KeyError("No key or prefix: %s" % key)
return result
def record(value=None):
"""This function returns a :class:`Record` instance constructed with an
initial value that you provide.
:param `value`: An initial record value.
:type `value`: ``dict``
"""
if value is None: value = {}
return Record(value)
dist/escu/bin/splunklib/modularinput/__init__.py
Vendored
-12
View File
@@ -1,12 +0,0 @@
"""The following imports allow these classes to be imported via
the splunklib.modularinput package like so:
from splunklib.modularinput import *
"""
from .argument import Argument
from .event import Event
from .event_writer import EventWriter
from .input_definition import InputDefinition
from .scheme import Scheme
from .script import Script
from .validation_definition import ValidationDefinition
dist/escu/bin/splunklib/modularinput/argument.py
Vendored
-103
View File
@@ -1,103 +0,0 @@
# Copyright 2011-2015 Splunk, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"): you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from __future__ import absolute_import
try:
import xml.etree.ElementTree as ET
except ImportError:
import xml.etree.cElementTree as ET
class Argument(object):
"""Class representing an argument to a modular input kind.
``Argument`` is meant to be used with ``Scheme`` to generate an XML
definition of the modular input kind that Splunk understands.
``name`` is the only required parameter for the constructor.
**Example with least parameters**::
arg1 = Argument(name="arg1")
**Example with all parameters**::
arg2 = Argument(
name="arg2",
description="This is an argument with lots of parameters",
validation="is_pos_int('some_name')",
data_type=Argument.data_type_number,
required_on_edit=True,
required_on_create=True
)
"""
# Constant values, do not change.
# These should be used for setting the value of an Argument object's data_type field.
data_type_boolean = "BOOLEAN"
data_type_number = "NUMBER"
data_type_string = "STRING"
def __init__(self, name, description=None, validation=None,
data_type=data_type_string, required_on_edit=False, required_on_create=False, title=None):
"""
:param name: ``string``, identifier for this argument in Splunk.
:param description: ``string``, human-readable description of the argument.
:param validation: ``string`` specifying how the argument should be validated, if using internal validation.
If using external validation, this will be ignored.
:param data_type: ``string``, data type of this field; use the class constants.
"data_type_boolean", "data_type_number", or "data_type_string".
:param required_on_edit: ``Boolean``, whether this arg is required when editing an existing modular input of this kind.
:param required_on_create: ``Boolean``, whether this arg is required when creating a modular input of this kind.
:param title: ``String``, a human-readable title for the argument.
"""
self.name = name
self.description = description
self.validation = validation
self.data_type = data_type
self.required_on_edit = required_on_edit
self.required_on_create = required_on_create
self.title = title
def add_to_document(self, parent):
"""Adds an ``Argument`` object to this ElementTree document.
Adds an <arg> subelement to the parent element, typically <args>
and sets up its subelements with their respective text.
:param parent: An ``ET.Element`` to be the parent of a new <arg> subelement
:returns: An ``ET.Element`` object representing this argument.
"""
arg = ET.SubElement(parent, "arg")
arg.set("name", self.name)
if self.title is not None:
ET.SubElement(arg, "title").text = self.title
if self.description is not None:
ET.SubElement(arg, "description").text = self.description
if self.validation is not None:
ET.SubElement(arg, "validation").text = self.validation
# add all other subelements to this Argument, represented by (tag, text)
subelements = [
("data_type", self.data_type),
("required_on_edit", self.required_on_edit),
("required_on_create", self.required_on_create)
]
for name, value in subelements:
ET.SubElement(arg, name).text = str(value).lower()
return arg
dist/escu/bin/splunklib/modularinput/event.py
Vendored
-114
View File
@@ -1,114 +0,0 @@
# Copyright 2011-2015 Splunk, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"): you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from __future__ import absolute_import
from io import TextIOBase
from splunklib.six import ensure_text
try:
import xml.etree.cElementTree as ET
except ImportError as ie:
import xml.etree.ElementTree as ET
class Event(object):
"""Represents an event or fragment of an event to be written by this modular input to Splunk.
To write an input to a stream, call the ``write_to`` function, passing in a stream.
"""
def __init__(self, data=None, stanza=None, time=None, host=None, index=None, source=None,
sourcetype=None, done=True, unbroken=True):
"""There are no required parameters for constructing an Event
**Example with minimal configuration**::
my_event = Event(
data="This is a test of my new event.",
stanza="myStanzaName",
time="%.3f" % 1372187084.000
)
**Example with full configuration**::
excellent_event = Event(
data="This is a test of my excellent event.",
stanza="excellenceOnly",
time="%.3f" % 1372274622.493,
host="localhost",
index="main",
source="Splunk",
sourcetype="misc",
done=True,
unbroken=True
)
:param data: ``string``, the event's text.
:param stanza: ``string``, name of the input this event should be sent to.
:param time: ``float``, time in seconds, including up to 3 decimal places to represent milliseconds.
:param host: ``string``, the event's host, ex: localhost.
:param index: ``string``, the index this event is specified to write to, or None if default index.
:param source: ``string``, the source of this event, or None to have Splunk guess.
:param sourcetype: ``string``, source type currently set on this event, or None to have Splunk guess.
:param done: ``boolean``, is this a complete ``Event``? False if an ``Event`` fragment.
:param unbroken: ``boolean``, Is this event completely encapsulated in this ``Event`` object?
"""
self.data = data
self.done = done
self.host = host
self.index = index
self.source = source
self.sourceType = sourcetype
self.stanza = stanza
self.time = time
self.unbroken = unbroken
def write_to(self, stream):
"""Write an XML representation of self, an ``Event`` object, to the given stream.
The ``Event`` object will only be written if its data field is defined,
otherwise a ``ValueError`` is raised.
:param stream: stream to write XML to.
"""
if self.data is None:
raise ValueError("Events must have at least the data field set to be written to XML.")
event = ET.Element("event")
if self.stanza is not None:
event.set("stanza", self.stanza)
event.set("unbroken", str(int(self.unbroken)))
# if a time isn't set, let Splunk guess by not creating a <time> element
if self.time is not None:
ET.SubElement(event, "time").text = str(self.time)
# add all other subelements to this Event, represented by (tag, text)
subelements = [
("source", self.source),
("sourcetype", self.sourceType),
("index", self.index),
("host", self.host),
("data", self.data)
]
for node, value in subelements:
if value is not None:
ET.SubElement(event, node).text = value
if self.done:
ET.SubElement(event, "done")
if isinstance(stream, TextIOBase):
stream.write(ensure_text(ET.tostring(event)))
else:
stream.write(ET.tostring(event))
stream.flush()
dist/escu/bin/splunklib/modularinput/event_writer.py
Vendored
-87
View File
@@ -1,87 +0,0 @@
# Copyright 2011-2015 Splunk, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"): you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from __future__ import absolute_import
import sys
from io import TextIOWrapper, TextIOBase
from splunklib.six import ensure_str
from .event import ET
try:
from splunklib.six.moves import cStringIO as StringIO
except ImportError:
from splunklib.six import StringIO
class EventWriter(object):
"""``EventWriter`` writes events and error messages to Splunk from a modular input.
Its two important methods are ``writeEvent``, which takes an ``Event`` object,
and ``log``, which takes a severity and an error message.
"""
# Severities that Splunk understands for log messages from modular inputs.
# Do not change these
DEBUG = "DEBUG"
INFO = "INFO"
WARN = "WARN"
ERROR = "ERROR"
FATAL = "FATAL"
def __init__(self, output = sys.stdout, error = sys.stderr):
"""
:param output: Where to write the output; defaults to sys.stdout.
:param error: Where to write any errors; defaults to sys.stderr.
"""
self._out = output
self._err = error
# has the opening <stream> tag been written yet?
self.header_written = False
def write_event(self, event):
"""Writes an ``Event`` object to Splunk.
:param event: An ``Event`` object.
"""
if not self.header_written:
self._out.write("<stream>")
self.header_written = True
event.write_to(self._out)
def log(self, severity, message):
"""Logs messages about the state of this modular input to Splunk.
These messages will show up in Splunk's internal logs.
:param severity: ``string``, severity of message, see severities defined as class constants.
:param message: ``string``, message to log.
"""
self._err.write("%s %s\n" % (severity, message))
self._err.flush()
def write_xml_document(self, document):
"""Writes a string representation of an
``ElementTree`` object to the output stream.
:param document: An ``ElementTree`` object.
"""
self._out.write(ensure_str(ET.tostring(document)))
self._out.flush()
def close(self):
"""Write the closing </stream> tag to make this XML well formed."""
self._out.write("</stream>")
self._out.flush()
dist/escu/bin/splunklib/modularinput/input_definition.py
Vendored
-60
View File
@@ -1,60 +0,0 @@
# Copyright 2011-2015 Splunk, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"): you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from __future__ import absolute_import
try:
import xml.etree.cElementTree as ET
except ImportError as ie:
import xml.etree.ElementTree as ET
from .utils import parse_xml_data
class InputDefinition:
"""``InputDefinition`` encodes the XML defining inputs that Splunk passes to
a modular input script.
**Example**::
i = InputDefinition()
"""
def __init__ (self):
self.metadata = {}
self.inputs = {}
def __eq__(self, other):
if not isinstance(other, InputDefinition):
return False
return self.metadata == other.metadata and self.inputs == other.inputs
@staticmethod
def parse(stream):
"""Parse a stream containing XML into an ``InputDefinition``.
:param stream: stream containing XML to parse.
:return: definition: an ``InputDefinition`` object.
"""
definition = InputDefinition()
# parse XML from the stream, then get the root node
root = ET.parse(stream).getroot()
for node in root:
if node.tag == "configuration":
# get config for each stanza
definition.inputs = parse_xml_data(node, "stanza")
else:
definition.metadata[node.tag] = node.text
return definition
dist/escu/bin/splunklib/modularinput/scheme.py
Vendored
-85
View File
@@ -1,85 +0,0 @@
# Copyright 2011-2015 Splunk, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"): you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from __future__ import absolute_import
try:
import xml.etree.cElementTree as ET
except ImportError:
import xml.etree.ElementTree as ET
class Scheme(object):
"""Class representing the metadata for a modular input kind.
A ``Scheme`` specifies a title, description, several options of how Splunk should run modular inputs of this
kind, and a set of arguments which define a particular modular input's properties.
The primary use of ``Scheme`` is to abstract away the construction of XML to feed to Splunk.
"""
# Constant values, do not change
# These should be used for setting the value of a Scheme object's streaming_mode field.
streaming_mode_simple = "SIMPLE"
streaming_mode_xml = "XML"
def __init__(self, title):
"""
:param title: ``string`` identifier for this Scheme in Splunk.
"""
self.title = title
self.description = None
self.use_external_validation = True
self.use_single_instance = False
self.streaming_mode = Scheme.streaming_mode_xml
# list of Argument objects, each to be represented by an <arg> tag
self.arguments = []
def add_argument(self, arg):
"""Add the provided argument, ``arg``, to the ``self.arguments`` list.
:param arg: An ``Argument`` object to add to ``self.arguments``.
"""
self.arguments.append(arg)
def to_xml(self):
"""Creates an ``ET.Element`` representing self, then returns it.
:returns: an ``ET.Element`` representing this scheme.
"""
root = ET.Element("scheme")
ET.SubElement(root, "title").text = self.title
# add a description subelement if it's defined
if self.description is not None:
ET.SubElement(root, "description").text = self.description
# add all other subelements to this Scheme, represented by (tag, text)
subelements = [
("use_external_validation", self.use_external_validation),
("use_single_instance", self.use_single_instance),
("streaming_mode", self.streaming_mode)
]
for name, value in subelements:
ET.SubElement(root, name).text = str(value).lower()
endpoint = ET.SubElement(root, "endpoint")
args = ET.SubElement(endpoint, "args")
# add arguments as subelements to the <args> element
for arg in self.arguments:
arg.add_to_document(args)
return root
dist/escu/bin/splunklib/modularinput/script.py
Vendored
-177
View File
@@ -1,177 +0,0 @@
# Copyright 2011-2015 Splunk, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"): you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from __future__ import absolute_import
from abc import ABCMeta, abstractmethod
from splunklib.six.moves.urllib.parse import urlsplit
import sys
from ..client import Service
from .event_writer import EventWriter
from .input_definition import InputDefinition
from .validation_definition import ValidationDefinition
from splunklib import six
try:
import xml.etree.cElementTree as ET
except ImportError:
import xml.etree.ElementTree as ET
class Script(six.with_metaclass(ABCMeta, object)):
"""An abstract base class for implementing modular inputs.
Subclasses should override ``get_scheme``, ``stream_events``,
and optionally ``validate_input`` if the modular input uses
external validation.
The ``run`` function is used to run modular inputs; it typically should
not be overridden.
"""
def __init__(self):
self._input_definition = None
self._service = None
def run(self, args):
"""Runs this modular input
:param args: List of command line arguments passed to this script.
:returns: An integer to be used as the exit value of this program.
"""
# call the run_script function, which handles the specifics of running
# a modular input
return self.run_script(args, EventWriter(), sys.stdin)
def run_script(self, args, event_writer, input_stream):
"""Handles all the specifics of running a modular input
:param args: List of command line arguments passed to this script.
:param event_writer: An ``EventWriter`` object for writing events.
:param input_stream: An input stream for reading inputs.
:returns: An integer to be used as the exit value of this program.
"""
try:
if len(args) == 1:
# This script is running as an input. Input definitions will be
# passed on stdin as XML, and the script will write events on
# stdout and log entries on stderr.
self._input_definition = InputDefinition.parse(input_stream)
self.stream_events(self._input_definition, event_writer)
event_writer.close()
return 0
elif str(args[1]).lower() == "--scheme":
# Splunk has requested XML specifying the scheme for this
# modular input Return it and exit.
scheme = self.get_scheme()
if scheme is None:
event_writer.log(
EventWriter.FATAL,
"Modular input script returned a null scheme.")
return 1
else:
event_writer.write_xml_document(scheme.to_xml())
return 0
elif args[1].lower() == "--validate-arguments":
validation_definition = ValidationDefinition.parse(input_stream)
try:
self.validate_input(validation_definition)
return 0
except Exception as e:
root = ET.Element("error")
ET.SubElement(root, "message").text = str(e)
event_writer.write_xml_document(root)
return 1
else:
err_string = "ERROR Invalid arguments to modular input script:" + ' '.join(
args)
event_writer._err.write(err_string)
return 1
except Exception as e:
event_writer.log(EventWriter.ERROR, str(e))
return 1
@property
def service(self):
""" Returns a Splunk service object for this script invocation.
The service object is created from the Splunkd URI and session key
passed to the command invocation on the modular input stream. It is
available as soon as the :code:`Script.stream_events` method is
called.
:return: :class:`splunklib.client.Service`. A value of None is returned,
if you call this method before the :code:`Script.stream_events` method
is called.
"""
if self._service is not None:
return self._service
if self._input_definition is None:
return None
splunkd_uri = self._input_definition.metadata["server_uri"]
session_key = self._input_definition.metadata["session_key"]
splunkd = urlsplit(splunkd_uri, allow_fragments=False)
self._service = Service(
scheme=splunkd.scheme,
host=splunkd.hostname,
port=splunkd.port,
token=session_key,
)
return self._service
@abstractmethod
def get_scheme(self):
"""The scheme defines the parameters understood by this modular input.
:return: a ``Scheme`` object representing the parameters for this modular input.
"""
def validate_input(self, definition):
"""Handles external validation for modular input kinds.
When Splunk calls a modular input script in validation mode, it will
pass in an XML document giving information about the Splunk instance (so
you can call back into it if needed) and the name and parameters of the
proposed input.
If this function does not throw an exception, the validation is assumed
to succeed. Otherwise any errors thrown will be turned into a string and
logged back to Splunk.
The default implementation always passes.
:param definition: The parameters for the proposed input passed by splunkd.
"""
pass
@abstractmethod
def stream_events(self, inputs, ew):
"""The method called to stream events into Splunk. It should do all of its output via
EventWriter rather than assuming that there is a console attached.
:param inputs: An ``InputDefinition`` object.
:param ew: An object with methods to write events and log messages to Splunk.
"""
dist/escu/bin/splunklib/modularinput/utils.py
Vendored
-74
View File
@@ -1,74 +0,0 @@
# Copyright 2011-2015 Splunk, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"): you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
# File for utility functions
from __future__ import absolute_import
from splunklib.six.moves import zip
def xml_compare(expected, found):
"""Checks equality of two ``ElementTree`` objects.
:param expected: An ``ElementTree`` object.
:param found: An ``ElementTree`` object.
:return: ``Boolean``, whether the two objects are equal.
"""
# if comparing the same ET object
if expected == found:
return True
# compare element attributes, ignoring order
if set(expected.items()) != set(found.items()):
return False
# check for equal number of children
expected_children = list(expected)
found_children = list(found)
if len(expected_children) != len(found_children):
return False
# compare children
if not all([xml_compare(a, b) for a, b in zip(expected_children, found_children)]):
return False
# compare elements, if there is no text node, return True
if (expected.text is None or expected.text.strip() == "") \
and (found.text is None or found.text.strip() == ""):
return True
else:
return expected.tag == found.tag and expected.text == found.text \
and expected.attrib == found.attrib
def parse_parameters(param_node):
if param_node.tag == "param":
return param_node.text
elif param_node.tag == "param_list":
parameters = []
for mvp in param_node:
parameters.append(mvp.text)
return parameters
else:
raise ValueError("Invalid configuration scheme, %s tag unexpected." % param_node.tag)
def parse_xml_data(parent_node, child_node_tag):
data = {}
for child in parent_node:
if child.tag == child_node_tag:
if child_node_tag == "stanza":
data[child.get("name")] = {}
for param in child:
data[child.get("name")][param.get("name")] = parse_parameters(param)
elif "item" == parent_node.tag:
data[child.get("name")] = parse_parameters(child)
return data
dist/escu/bin/splunklib/modularinput/validation_definition.py
Vendored
-86
View File
@@ -1,86 +0,0 @@
# Copyright 2011-2015 Splunk, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"): you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from __future__ import absolute_import
try:
import xml.etree.cElementTree as ET
except ImportError as ie:
import xml.etree.ElementTree as ET
from .utils import parse_xml_data
class ValidationDefinition(object):
"""This class represents the XML sent by Splunk for external validation of a
new modular input.
**Example**::
v = ValidationDefinition()
"""
def __init__(self):
self.metadata = {}
self.parameters = {}
def __eq__(self, other):
if not isinstance(other, ValidationDefinition):
return False
return self.metadata == other.metadata and self.parameters == other.parameters
@staticmethod
def parse(stream):
"""Creates a ``ValidationDefinition`` from a provided stream containing XML.
The XML typically will look like this:
.. code-block:: xml
<items>
<server_host>myHost</server_host>
<server_uri>https://127.0.0.1:8089</server_uri>
<session_key>123102983109283019283</session_key>
<checkpoint_dir>/opt/splunk/var/lib/splunk/modinputs</checkpoint_dir>
<item name="myScheme">
<param name="param1">value1</param>
<param_list name="param2">
<value>value2</value>
<value>value3</value>
<value>value4</value>
</param_list>
</item>
</items>
:param stream: ``Stream`` containing XML to parse.
:return: A ``ValidationDefinition`` object.
"""
definition = ValidationDefinition()
# parse XML from the stream, then get the root node
root = ET.parse(stream).getroot()
for node in root:
# lone item node
if node.tag == "item":
# name from item node
definition.metadata["name"] = node.get("name")
definition.parameters = parse_xml_data(node, "")
else:
# Store anything else in metadata
definition.metadata[node.tag] = node.text
return definition
dist/escu/bin/splunklib/ordereddict.py
Vendored
-128
View File
@@ -1,128 +0,0 @@
# Copyright (c) 2009 Raymond Hettinger
#
# Permission is hereby granted, free of charge, to any person
# obtaining a copy of this software and associated documentation files
# (the "Software"), to deal in the Software without restriction,
# including without limitation the rights to use, copy, modify, merge,
# publish, distribute, sublicense, and/or sell copies of the Software,
# and to permit persons to whom the Software is furnished to do so,
# subject to the following conditions:
#
# The above copyright notice and this permission notice shall be
# included in all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
# OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
# HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
# WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
# OTHER DEALINGS IN THE SOFTWARE.
from UserDict import DictMixin
class OrderedDict(dict, DictMixin):
def __init__(self, *args, **kwds):
if len(args) > 1:
raise TypeError('expected at most 1 arguments, got %d' % len(args))
try:
self.__end
except AttributeError:
self.clear()
self.update(*args, **kwds)
def clear(self):
self.__end = end = []
end += [None, end, end] # sentinel node for doubly linked list
self.__map = {} # key --> [key, prev, next]
dict.clear(self)
def __setitem__(self, key, value):
if key not in self:
end = self.__end
curr = end[1]
curr[2] = end[1] = self.__map[key] = [key, curr, end]
dict.__setitem__(self, key, value)
def __delitem__(self, key):
dict.__delitem__(self, key)
key, prev, next = self.__map.pop(key)
prev[2] = next
next[1] = prev
def __iter__(self):
end = self.__end
curr = end[2]
while curr is not end:
yield curr[0]
curr = curr[2]
def __reversed__(self):
end = self.__end
curr = end[1]
while curr is not end:
yield curr[0]
curr = curr[1]
def popitem(self, last=True):
if not self:
raise KeyError('dictionary is empty')
if last:
key = reversed(self).next()
else:
key = iter(self).next()
value = self.pop(key)
return key, value
def __reduce__(self):
items = [[k, self[k]] for k in self]
tmp = self.__map, self.__end
del self.__map, self.__end
inst_dict = vars(self).copy()
self.__map, self.__end = tmp
if inst_dict:
return (self.__class__, (items,), inst_dict)
return self.__class__, (items,)
def keys(self):
return list(self)
setdefault = DictMixin.setdefault
update = DictMixin.update
pop = DictMixin.pop
values = DictMixin.values
items = DictMixin.items
iterkeys = DictMixin.iterkeys
itervalues = DictMixin.itervalues
iteritems = DictMixin.iteritems
def __repr__(self):
if not self:
return '%s()' % (self.__class__.__name__,)
return '%s(%r)' % (self.__class__.__name__, self.items())
def copy(self):
return self.__class__(self)
@classmethod
def fromkeys(cls, iterable, value=None):
d = cls()
for key in iterable:
d[key] = value
return d
def __eq__(self, other):
if isinstance(other, OrderedDict):
if len(self) != len(other):
return False
for p, q in zip(self.items(), other.items()):
if p != q:
return False
return True
return dict.__eq__(self, other)
def __ne__(self, other):
return not self == other
dist/escu/bin/splunklib/results.py
Vendored
-295
View File
@@ -1,295 +0,0 @@
# Copyright 2011-2015 Splunk, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"): you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
"""The **splunklib.results** module provides a streaming XML reader for Splunk
search results.
Splunk search results can be returned in a variety of formats including XML,
JSON, and CSV. To make it easier to stream search results in XML format, they
are returned as a stream of XML *fragments*, not as a single XML document. This
module supports incrementally reading one result record at a time from such a
result stream. This module also provides a friendly iterator-based interface for
accessing search results while avoiding buffering the result set, which can be
very large.
To use the reader, instantiate :class:`ResultsReader` on a search result stream
as follows:::
reader = ResultsReader(result_stream)
for item in reader:
print(item)
print "Results are a preview: %s" % reader.is_preview
"""
from __future__ import absolute_import
from io import BytesIO
from splunklib import six
try:
import xml.etree.cElementTree as et
except:
import xml.etree.ElementTree as et
try:
from collections import OrderedDict # must be python 2.7
except ImportError:
from .ordereddict import OrderedDict
try:
from splunklib.six.moves import cStringIO as StringIO
except:
from splunklib.six import StringIO
__all__ = [
"ResultsReader",
"Message"
]
class Message(object):
"""This class represents informational messages that Splunk interleaves in the results stream.
``Message`` takes two arguments: a string giving the message type (e.g., "DEBUG"), and
a string giving the message itself.
**Example**::
m = Message("DEBUG", "There's something in that variable...")
"""
def __init__(self, type_, message):
self.type = type_
self.message = message
def __repr__(self):
return "%s: %s" % (self.type, self.message)
def __eq__(self, other):
return (self.type, self.message) == (other.type, other.message)
def __hash__(self):
return hash((self.type, self.message))
class _ConcatenatedStream(object):
"""Lazily concatenate zero or more streams into a stream.
As you read from the concatenated stream, you get characters from
each stream passed to ``_ConcatenatedStream``, in order.
**Example**::
from StringIO import StringIO
s = _ConcatenatedStream(StringIO("abc"), StringIO("def"))
assert s.read() == "abcdef"
"""
def __init__(self, *streams):
self.streams = list(streams)
def read(self, n=None):
"""Read at most *n* characters from this stream.
If *n* is ``None``, return all available characters.
"""
response = b""
while len(self.streams) > 0 and (n is None or n > 0):
txt = self.streams[0].read(n)
response += txt
if n is not None:
n -= len(txt)
if n is None or n > 0:
del self.streams[0]
return response
class _XMLDTDFilter(object):
"""Lazily remove all XML DTDs from a stream.
All substrings matching the regular expression <?[^>]*> are
removed in their entirety from the stream. No regular expressions
are used, however, so everything still streams properly.
**Example**::
from StringIO import StringIO
s = _XMLDTDFilter("<?xml abcd><element><?xml ...></element>")
assert s.read() == "<element></element>"
"""
def __init__(self, stream):
self.stream = stream
def read(self, n=None):
"""Read at most *n* characters from this stream.
If *n* is ``None``, return all available characters.
"""
response = b""
while n is None or n > 0:
c = self.stream.read(1)
if c == b"":
break
elif c == b"<":
c += self.stream.read(1)
if c == b"<?":
while True:
q = self.stream.read(1)
if q == b">":
break
else:
response += c
if n is not None:
n -= len(c)
else:
response += c
if n is not None:
n -= 1
return response
class ResultsReader(object):
"""This class returns dictionaries and Splunk messages from an XML results
stream.
``ResultsReader`` is iterable, and returns a ``dict`` for results, or a
:class:`Message` object for Splunk messages. This class has one field,
``is_preview``, which is ``True`` when the results are a preview from a
running search, or ``False`` when the results are from a completed search.
This function has no network activity other than what is implicit in the
stream it operates on.
:param `stream`: The stream to read from (any object that supports
``.read()``).
**Example**::
import results
response = ... # the body of an HTTP response
reader = results.ResultsReader(response)
for result in reader:
if isinstance(result, dict):
print "Result: %s" % result
elif isinstance(result, results.Message):
print "Message: %s" % result
print "is_preview = %s " % reader.is_preview
"""
# Be sure to update the docstrings of client.Jobs.oneshot,
# client.Job.results_preview and client.Job.results to match any
# changes made to ResultsReader.
#
# This wouldn't be a class, just the _parse_results function below,
# except that you cannot get the current generator inside the
# function creating that generator. Thus it's all wrapped up for
# the sake of one field.
def __init__(self, stream):
# The search/jobs/exports endpoint, when run with
# earliest_time=rt and latest_time=rt streams a sequence of
# XML documents, each containing a result, as opposed to one
# results element containing lots of results. Python's XML
# parsers are broken, and instead of reading one full document
# and returning the stream that follows untouched, they
# destroy the stream and throw an error. To get around this,
# we remove all the DTD definitions inline, then wrap the
# fragments in a fiction <doc> element to make the parser happy.
stream = _XMLDTDFilter(stream)
stream = _ConcatenatedStream(BytesIO(b"<doc>"), stream, BytesIO(b"</doc>"))
self.is_preview = None
self._gen = self._parse_results(stream)
def __iter__(self):
return self
def next(self):
return next(self._gen)
__next__ = next
def _parse_results(self, stream):
"""Parse results and messages out of *stream*."""
result = None
values = None
try:
for event, elem in et.iterparse(stream, events=('start', 'end')):
if elem.tag == 'results' and event == 'start':
# The wrapper element is a <results preview="0|1">. We
# don't care about it except to tell is whether these
# are preview results, or the final results from the
# search.
is_preview = elem.attrib['preview'] == '1'
self.is_preview = is_preview
if elem.tag == 'result':
if event == 'start':
result = OrderedDict()
elif event == 'end':
yield result
result = None
elem.clear()
elif elem.tag == 'field' and result is not None:
# We need the 'result is not None' check because
# 'field' is also the element name in the <meta>
# header that gives field order, which is not what we
# want at all.
if event == 'start':
values = []
elif event == 'end':
field_name = elem.attrib['k']
if len(values) == 1:
result[field_name] = values[0]
else:
result[field_name] = values
# Calling .clear() is necessary to let the
# element be garbage collected. Otherwise
# arbitrarily large results sets will use
# arbitrarily large memory intead of
# streaming.
elem.clear()
elif elem.tag in ('text', 'v') and event == 'end':
try:
text = "".join(elem.itertext())
except AttributeError:
# Assume we're running in Python < 2.7, before itertext() was added
# So we'll define it here
def __itertext(self):
tag = self.tag
if not isinstance(tag, six.string_types) and tag is not None:
return
if self.text:
yield self.text
for e in self:
for s in __itertext(e):
yield s
if e.tail:
yield e.tail
text = "".join(__itertext(elem))
values.append(text)
elem.clear()
elif elem.tag == 'msg':
if event == 'start':
msg_type = elem.attrib['type']
elif event == 'end':
text = elem.text if elem.text is not None else ""
yield Message(msg_type, text)
elem.clear()
except SyntaxError as pe:
# This is here to handle the same incorrect return from
# splunk that is described in __init__.
if 'no element found' in pe.msg:
return
else:
raise
dist/escu/bin/splunklib/searchcommands/__init__.py
Vendored
-155
View File
@@ -1,155 +0,0 @@
# coding=utf-8
#
# Copyright © 2011-2015 Splunk, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"): you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
"""
.. topic:: Design Notes
1. Commands are constrained to this ABNF grammar::
command = command-name *[wsp option] *[wsp [dquote] field-name [dquote]]
command-name = alpha *( alpha / digit )
option = option-name [wsp] "=" [wsp] option-value
option-name = alpha *( alpha / digit / "_" )
option-value = word / quoted-string
word = 1*( %01-%08 / %0B / %0C / %0E-1F / %21 / %23-%FF ) ; Any character but DQUOTE and WSP
quoted-string = dquote *( word / wsp / "\" dquote / dquote dquote ) dquote
field-name = ( "_" / alpha ) *( alpha / digit / "_" / "." / "-" )
It does not show that :code:`field-name` values may be comma-separated. This is because Splunk strips commas from
the command line. A search command will never see them.
2. Search commands targeting versions of Splunk prior to 6.3 must be statically configured as follows:
.. code-block:: text
:linenos:
[command_name]
filename = command_name.py
supports_getinfo = true
supports_rawargs = true
No other static configuration is required or expected and may interfere with command execution.
3. Commands support dynamic probing for settings.
Splunk probes for settings dynamically when :code:`supports_getinfo=true`.
You must add this line to the commands.conf stanza for each of your search
commands.
4. Commands do not support parsed arguments on the command line.
Splunk parses arguments when :code:`supports_rawargs=false`. The
:code:`SearchCommand` class sets this value unconditionally. You cannot
override it.
**Rationale**
Splunk parses arguments by stripping quotes, nothing more. This may be useful
in some cases, but doesn't work well with our chosen grammar.
5. Commands consume input headers.
An input header is provided by Splunk when :code:`enableheader=true`. The
:class:`SearchCommand` class sets this value unconditionally. You cannot
override it.
6. Commands produce an output messages header.
Splunk expects a command to produce an output messages header when
:code:`outputheader=true`. The :class:`SearchCommand` class sets this value
unconditionally. You cannot override it.
7. Commands support multi-value fields.
Multi-value fields are provided and consumed by Splunk when
:code:`supports_multivalue=true`. This value is fixed. You cannot override
it.
8. This module represents all fields on the output stream in multi-value
format.
Splunk recognizes two kinds of data: :code:`value` and :code:`list(value)`.
The multi-value format represents these data in field pairs. Given field
:code:`name` the multi-value format calls for the creation of this pair of
fields.
================= =========================================================
Field name Field data
================= =========================================================
:code:`name` Value or text from which a list of values was derived.
:code:`__mv_name` Empty, if :code:`field` represents a :code:`value`;
otherwise, an encoded :code:`list(value)`. Values in the
list are wrapped in dollar signs ($) and separated by
semi-colons (;). Dollar signs ($) within a value are
represented by a pair of dollar signs ($$).
================= =========================================================
Serializing data in this format enables streaming and reduces a command's
memory footprint at the cost of one extra byte of data per field per record
and a small amount of extra processing time by the next command in the
pipeline.
9. A :class:`ReportingCommand` must override :meth:`~ReportingCommand.reduce`
and may override :meth:`~ReportingCommand.map`. Map/reduce commands on the
Splunk processing pipeline are distinguished as this example illustrates.
**Splunk command**
.. code-block:: text
sum total=total_date_hour date_hour
**Map command line**
.. code-block:: text
sum __GETINFO__ __map__ total=total_date_hour date_hour
sum __EXECUTE__ __map__ total=total_date_hour date_hour
**Reduce command line**
.. code-block:: text
sum __GETINFO__ total=total_date_hour date_hour
sum __EXECUTE__ total=total_date_hour date_hour
The :code:`__map__` argument is introduced by
:meth:`ReportingCommand._execute`. Search command authors cannot influence
the contents of the command line in this release.
.. topic:: References
1. `Search command style guide <http://docs.splunk.com/Documentation/Splunk/6.0/Search/Searchcommandstyleguide>`__
2. `Commands.conf.spec <http://docs.splunk.com/Documentation/Splunk/5.0.5/Admin/Commandsconf>`_
"""
from __future__ import absolute_import, division, print_function, unicode_literals
from .environment import *
from .decorators import *
from .validators import *
from .generating_command import GeneratingCommand
from .streaming_command import StreamingCommand
from .eventing_command import EventingCommand
from .reporting_command import ReportingCommand
from .external_search_command import execute, ExternalSearchCommand
from .search_command import dispatch, SearchMetric
dist/escu/bin/splunklib/searchcommands/decorators.py
Vendored
-450
View File
@@ -1,450 +0,0 @@
# coding=utf-8
#
# Copyright © 2011-2015 Splunk, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"): you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from __future__ import absolute_import, division, print_function, unicode_literals
from splunklib import six
try:
from collections import OrderedDict # must be python 2.7
except ImportError:
from ..ordereddict import OrderedDict
from inspect import getmembers, isclass, isfunction
from splunklib.six.moves import map as imap
from .internals import ConfigurationSettingsType, json_encode_string
from .validators import OptionName
class Configuration(object):
""" Defines the configuration settings for a search command.
Documents, validates, and ensures that only relevant configuration settings are applied. Adds a :code:`name` class
variable to search command classes that don't have one. The :code:`name` is derived from the name of the class.
By convention command class names end with the word "Command". To derive :code:`name` the word "Command" is removed
from the end of the class name and then converted to lower case for conformance with the `Search command style guide
<http://docs.splunk.com/Documentation/Splunk/latest/Search/Searchcommandstyleguide>`__
"""
def __init__(self, o=None, **kwargs):
#
# The o argument enables the configuration decorator to be used with or without parentheses. For example, it
# enables you to write code that looks like this:
#
# @Configuration
# class Foo(SearchCommand):
# ...
#
# @Configuration()
# class Bar(SearchCommand):
# ...
#
# Without the o argument, the Python compiler will complain about the first form. With the o argument, both
# forms work. The first form provides a value for o: Foo. The second form does does not provide a value for o.
# The class or method decorated is not passed to the constructor. A value of None is passed instead.
#
self.settings = kwargs
def __call__(self, o):
if isfunction(o):
# We must wait to finalize configuration as the class containing this function is under construction
# at the time this call to decorate a member function. This will be handled in the call to
# o.ConfigurationSettings.fix_up(o) in the elif clause of this code block.
o._settings = self.settings
elif isclass(o):
# Set command name
name = o.__name__
if name.endswith('Command'):
name = name[:-len('Command')]
o.name = six.text_type(name.lower())
# Construct ConfigurationSettings instance for the command class
o.ConfigurationSettings = ConfigurationSettingsType(
module=o.__module__ + '.' + o.__name__,
name='ConfigurationSettings',
bases=(o.ConfigurationSettings,))
ConfigurationSetting.fix_up(o.ConfigurationSettings, self.settings)
o.ConfigurationSettings.fix_up(o)
Option.fix_up(o)
else:
raise TypeError('Incorrect usage: Configuration decorator applied to {0}'.format(type(o), o.__name__))
return o
class ConfigurationSetting(property):
""" Generates a :class:`property` representing the named configuration setting
This is a convenience function designed to reduce the amount of boiler-plate code you must write; most notably for
property setters.
:param name: Configuration setting name.
:type name: str or unicode
:param doc: A documentation string.
:type doc: bytes, unicode or NoneType
:param readonly: If true, specifies that the configuration setting is fixed.
:type name: bool or NoneType
:param value: Configuration setting value.
:return: A :class:`property` instance representing the configuration setting.
:rtype: property
"""
def __init__(self, fget=None, fset=None, fdel=None, doc=None, name=None, readonly=None, value=None):
property.__init__(self, fget=fget, fset=fset, fdel=fdel, doc=doc)
self._readonly = readonly
self._value = value
self._name = name
def __call__(self, function):
return self.getter(function)
def deleter(self, function):
return self._copy_extra_attributes(property.deleter(self, function))
def getter(self, function):
return self._copy_extra_attributes(property.getter(self, function))
def setter(self, function):
return self._copy_extra_attributes(property.setter(self, function))
@staticmethod
def fix_up(cls, values):
is_configuration_setting = lambda attribute: isinstance(attribute, ConfigurationSetting)
definitions = getmembers(cls, is_configuration_setting)
i = 0
for name, setting in definitions:
if setting._name is None:
setting._name = name = six.text_type(name)
else:
name = setting._name
validate, specification = setting._get_specification()
backing_field_name = '_' + name
if setting.fget is None and setting.fset is None and setting.fdel is None:
value = setting._value
if setting._readonly or value is not None:
validate(specification, name, value)
def fget(bfn, value):
return lambda this: getattr(this, bfn, value)
setting = setting.getter(fget(backing_field_name, value))
if not setting._readonly:
def fset(bfn, validate, specification, name):
return lambda this, value: setattr(this, bfn, validate(specification, name, value))
setting = setting.setter(fset(backing_field_name, validate, specification, name))
setattr(cls, name, setting)
def is_supported_by_protocol(supporting_protocols):
def is_supported_by_protocol(version):
return version in supporting_protocols
return is_supported_by_protocol
del setting._name, setting._value, setting._readonly
setting.is_supported_by_protocol = is_supported_by_protocol(specification.supporting_protocols)
setting.supporting_protocols = specification.supporting_protocols
setting.backing_field_name = backing_field_name
definitions[i] = setting
setting.name = name
i += 1
try:
value = values[name]
except KeyError:
continue
if setting.fset is None:
raise ValueError('The value of configuration setting {} is fixed'.format(name))
setattr(cls, backing_field_name, validate(specification, name, value))
del values[name]
if len(values) > 0:
settings = sorted(list(six.iteritems(values)))
settings = imap(lambda n_v: '{}={}'.format(n_v[0], repr(n_v[1])), settings)
raise AttributeError('Inapplicable configuration settings: ' + ', '.join(settings))
cls.configuration_setting_definitions = definitions
def _copy_extra_attributes(self, other):
other._readonly = self._readonly
other._value = self._value
other._name = self._name
return other
def _get_specification(self):
name = self._name
try:
specification = ConfigurationSettingsType.specification_matrix[name]
except KeyError:
raise AttributeError('Unknown configuration setting: {}={}'.format(name, repr(self._value)))
return ConfigurationSettingsType.validate_configuration_setting, specification
class Option(property):
""" Represents a search command option.
Required options must be specified on the search command line.
**Example:**
Short form (recommended). When you are satisfied with built-in or custom validation behaviors.
.. code-block:: python
:linenos:
from splunklib.searchcommands.decorators import Option
from splunklib.searchcommands.validators import Fieldname
total = Option(
doc=''' **Syntax:** **total=***<fieldname>*
**Description:** Name of the field that will hold the computed
sum''',
require=True, validate=Fieldname())
**Example:**
Long form. Useful when you wish to manage the option value and its deleter/getter/setter side-effects yourself. You
must provide a getter and a setter. If your :code:`Option` requires `destruction <https://docs.python.org/2/reference/datamodel.html#object.__del__>`_ you must
also provide a deleter. You must be prepared to accept a value of :const:`None` which indicates that your
:code:`Option` is unset.
.. code-block:: python
:linenos:
from splunklib.searchcommands import Option
@Option()
def logging_configuration(self):
\""" **Syntax:** logging_configuration=<path>
**Description:** Loads an alternative logging configuration file for a command invocation. The logging
configuration file must be in Python ConfigParser-format. The *<path>* name and all path names specified in
configuration are relative to the app root directory.
\"""
return self._logging_configuration
@logging_configuration.setter
def logging_configuration(self, value):
if value is not None
logging.configure(value)
self._logging_configuration = value
def __init__(self)
self._logging_configuration = None
"""
def __init__(self, fget=None, fset=None, fdel=None, doc=None, name=None, default=None, require=None, validate=None):
property.__init__(self, fget, fset, fdel, doc)
self.name = name
self.default = default
self.validate = validate
self.require = bool(require)
def __call__(self, function):
return self.getter(function)
# region Methods
def deleter(self, function):
return self._copy_extra_attributes(property.deleter(self, function))
def getter(self, function):
return self._copy_extra_attributes(property.getter(self, function))
def setter(self, function):
return self._copy_extra_attributes(property.setter(self, function))
@classmethod
def fix_up(cls, command_class):
is_option = lambda attribute: isinstance(attribute, Option)
definitions = getmembers(command_class, is_option)
validate_option_name = OptionName()
i = 0
for name, option in definitions:
if option.name is None:
option.name = name # no validation required
else:
validate_option_name(option.name)
if option.fget is None and option.fset is None and option.fdel is None:
backing_field_name = '_' + name
def fget(bfn):
return lambda this: getattr(this, bfn, None)
option = option.getter(fget(backing_field_name))
def fset(bfn, validate):
if validate is None:
return lambda this, value: setattr(this, bfn, value)
return lambda this, value: setattr(this, bfn, validate(value))
option = option.setter(fset(backing_field_name, option.validate))
setattr(command_class, name, option)
elif option.validate is not None:
def fset(function, validate):
return lambda this, value: function(this, validate(value))
option = option.setter(fset(option.fset, option.validate))
setattr(command_class, name, option)
definitions[i] = name, option
i += 1
command_class.option_definitions = definitions
def _copy_extra_attributes(self, other):
other.name = self.name
other.default = self.default
other.require = self.require
other.validate = self.validate
return other
# endregion
# region Types
class Item(object):
""" Presents an instance/class view over a search command `Option`.
This class is used by SearchCommand.process to parse and report on option values.
"""
def __init__(self, command, option):
self._command = command
self._option = option
self._is_set = False
validator = self.validator
self._format = six.text_type if validator is None else validator.format
def __repr__(self):
return '(' + repr(self.name) + ', ' + repr(self._format(self.value)) + ')'
def __str__(self):
value = self.value
value = 'None' if value is None else json_encode_string(self._format(value))
return self.name + '=' + value
# region Properties
@property
def is_required(self):
return bool(self._option.require)
@property
def is_set(self):
""" Indicates whether an option value was provided as argument.
"""
return self._is_set
@property
def name(self):
return self._option.name
@property
def validator(self):
return self._option.validate
@property
def value(self):
return self._option.__get__(self._command)
@value.setter
def value(self, value):
self._option.__set__(self._command, value)
self._is_set = True
# endregion
# region Methods
def reset(self):
self._option.__set__(self._command, self._option.default)
self._is_set = False
pass
# endregion
class View(OrderedDict):
""" Presents an ordered dictionary view of the set of :class:`Option` arguments to a search command.
This class is used by SearchCommand.process to parse and report on option values.
"""
def __init__(self, command):
definitions = type(command).option_definitions
item_class = Option.Item
OrderedDict.__init__(self, ((option.name, item_class(command, option)) for (name, option) in definitions))
def __repr__(self):
text = 'Option.View([' + ','.join(imap(lambda item: repr(item), six.itervalues(self))) + '])'
return text
def __str__(self):
text = ' '.join([str(item) for item in six.itervalues(self) if item.is_set])
return text
# region Methods
def get_missing(self):
missing = [item.name for item in six.itervalues(self) if item.is_required and not item.is_set]
return missing if len(missing) > 0 else None
def reset(self):
for value in six.itervalues(self):
value.reset()
pass
# endregion
pass
# endregion
__all__ = ['Configuration', 'Option']
dist/escu/bin/splunklib/searchcommands/environment.py
Vendored
-123
View File
@@ -1,123 +0,0 @@
# coding=utf-8
#
# Copyright © 2011-2015 Splunk, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"): you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from __future__ import absolute_import, division, print_function, unicode_literals
from logging import getLogger, root, StreamHandler
from logging.config import fileConfig
from os import chdir, environ, path
from splunklib.six.moves import getcwd
import sys
def configure_logging(logger_name, filename=None):
""" Configure logging and return the named logger and the location of the logging configuration file loaded.
This function expects a Splunk app directory structure::
<app-root>
bin
...
default
...
local
...
This function looks for a logging configuration file at each of these locations, loading the first, if any,
logging configuration file that it finds::
local/{name}.logging.conf
default/{name}.logging.conf
local/logging.conf
default/logging.conf
The current working directory is set to *<app-root>* before the logging configuration file is loaded. Hence, paths
in the logging configuration file are relative to *<app-root>*. The current directory is reset before return.
You may short circuit the search for a logging configuration file by providing an alternative file location in
`path`. Logging configuration files must be in `ConfigParser format`_.
#Arguments:
:param logger_name: Logger name
:type logger_name: bytes, unicode
:param filename: Location of an alternative logging configuration file or `None`.
:type filename: bytes, unicode or NoneType
:returns: The named logger and the location of the logging configuration file loaded.
:rtype: tuple
.. _ConfigParser format: https://docs.python.org/2/library/logging.config.html#configuration-file-format
"""
if filename is None:
if logger_name is None:
probing_paths = [path.join('local', 'logging.conf'), path.join('default', 'logging.conf')]
else:
probing_paths = [
path.join('local', logger_name + '.logging.conf'),
path.join('default', logger_name + '.logging.conf'),
path.join('local', 'logging.conf'),
path.join('default', 'logging.conf')]
for relative_path in probing_paths:
configuration_file = path.join(app_root, relative_path)
if path.exists(configuration_file):
filename = configuration_file
break
elif not path.isabs(filename):
found = False
for conf in 'local', 'default':
configuration_file = path.join(app_root, conf, filename)
if path.exists(configuration_file):
filename = configuration_file
found = True
break
if not found:
raise ValueError('Logging configuration file "{}" not found in local or default directory'.format(filename))
elif not path.exists(filename):
raise ValueError('Logging configuration file "{}" not found'.format(filename))
if filename is not None:
global _current_logging_configuration_file
filename = path.realpath(filename)
if filename != _current_logging_configuration_file:
working_directory = getcwd()
chdir(app_root)
try:
fileConfig(filename, {'SPLUNK_HOME': splunk_home})
finally:
chdir(working_directory)
_current_logging_configuration_file = filename
if len(root.handlers) == 0:
root.addHandler(StreamHandler())
return None if logger_name is None else getLogger(logger_name), filename
_current_logging_configuration_file = None
splunk_home = path.abspath(path.join(getcwd(), environ.get('SPLUNK_HOME', '')))
app_file = getattr(sys.modules['__main__'], '__file__', sys.executable)
app_root = path.dirname(path.abspath(path.dirname(app_file)))
splunklib_logger, logging_configuration = configure_logging('splunklib')
__all__ = ['app_file', 'app_root', 'logging_configuration', 'splunk_home', 'splunklib_logger']
dist/escu/bin/splunklib/searchcommands/eventing_command.py
Vendored
-149
View File
@@ -1,149 +0,0 @@
# coding=utf-8
#
# Copyright 2011-2015 Splunk, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"): you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from __future__ import absolute_import, division, print_function, unicode_literals
from splunklib import six
from splunklib.six.moves import map as imap
from .decorators import ConfigurationSetting
from .search_command import SearchCommand
class EventingCommand(SearchCommand):
""" Applies a transformation to search results as they travel through the events pipeline.
Eventing commands typically filter, group, order, and/or or augment event records. Examples of eventing commands
from Splunk's built-in command set include sort_, dedup_, and cluster_. Each execution of an eventing command
should produce a set of event records that is independently usable by downstream processors.
.. _sort: http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Sort
.. _dedup: http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Dedup
.. _cluster: http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Cluster
EventingCommand configuration
==============================
You can configure your command for operation under Search Command Protocol (SCP) version 1 or 2. SCP 2 requires
Splunk 6.3 or later.
"""
# region Methods
def transform(self, records):
""" Generator function that processes and yields event records to the Splunk events pipeline.
You must override this method.
"""
raise NotImplementedError('EventingCommand.transform(self, records)')
def _execute(self, ifile, process):
SearchCommand._execute(self, ifile, self.transform)
# endregion
class ConfigurationSettings(SearchCommand.ConfigurationSettings):
""" Represents the configuration settings that apply to a :class:`EventingCommand`.
"""
# region SCP v1/v2 properties
required_fields = ConfigurationSetting(doc='''
List of required fields for this search which back-propagates to the generating search.
Setting this value enables selected fields mode under SCP 2. Under SCP 1 you must also specify
:code:`clear_required_fields=True` to enable selected fields mode. To explicitly select all fields,
specify a value of :const:`['*']`. No error is generated if a specified field is missing.
Default: :const:`None`, which implicitly selects all fields.
''')
# endregion
# region SCP v1 properties
clear_required_fields = ConfigurationSetting(doc='''
:const:`True`, if required_fields represent the *only* fields required.
If :const:`False`, required_fields are additive to any fields that may be required by subsequent commands.
In most cases, :const:`False` is appropriate for eventing commands.
Default: :const:`False`
''')
retainsevents = ConfigurationSetting(readonly=True, value=True, doc='''
:const:`True`, if the command retains events the way the sort/dedup/cluster commands do.
If :const:`False`, the command transforms events the way the stats command does.
Fixed: :const:`True`
''')
# endregion
# region SCP v2 properties
maxinputs = ConfigurationSetting(doc='''
Specifies the maximum number of events that can be passed to the command for each invocation.
This limit cannot exceed the value of `maxresultrows` as defined in limits.conf_. Under SCP 1 you must
specify this value in commands.conf_.
Default: The value of `maxresultrows`.
Supported by: SCP 2
.. _limits.conf: http://docs.splunk.com/Documentation/Splunk/latest/admin/Limitsconf
''')
type = ConfigurationSetting(readonly=True, value='events', doc='''
Command type
Fixed: :const:`'events'`.
Supported by: SCP 2
''')
# endregion
# region Methods
@classmethod
def fix_up(cls, command):
""" Verifies :code:`command` class structure.
"""
if command.transform == EventingCommand.transform:
raise AttributeError('No EventingCommand.transform override')
SearchCommand.ConfigurationSettings.fix_up(command)
# TODO: Stop looking like a dictionary because we don't obey the semantics
# N.B.: Does not use Python 2 dict copy semantics
def iteritems(self):
iteritems = SearchCommand.ConfigurationSettings.iteritems(self)
return imap(lambda name_value: (name_value[0], 'events' if name_value[0] == 'type' else name_value[1]), iteritems)
# N.B.: Does not use Python 3 dict view semantics
if not six.PY2:
items = iteritems
# endregion
dist/escu/bin/splunklib/searchcommands/external_search_command.py
Vendored
-228
View File
@@ -1,228 +0,0 @@
# coding=utf-8
#
# Copyright 2011-2015 Splunk, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"): you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from __future__ import absolute_import, division, print_function, unicode_literals
from logging import getLogger
import os
import sys
import traceback
from splunklib import six
if sys.platform == 'win32':
from signal import signal, CTRL_BREAK_EVENT, SIGBREAK, SIGINT, SIGTERM
from subprocess import Popen
import atexit
from . import splunklib_logger as logger
# P1 [ ] TODO: Add ExternalSearchCommand class documentation
class ExternalSearchCommand(object):
"""
"""
def __init__(self, path, argv=None, environ=None):
if not isinstance(path, (bytes, six.text_type)):
raise ValueError('Expected a string value for path, not {}'.format(repr(path)))
self._logger = getLogger(self.__class__.__name__)
self._path = six.text_type(path)
self._argv = None
self._environ = None
self.argv = argv
self.environ = environ
# region Properties
@property
def argv(self):
return getattr(self, '_argv')
@argv.setter
def argv(self, value):
if not (value is None or isinstance(value, (list, tuple))):
raise ValueError('Expected a list, tuple or value of None for argv, not {}'.format(repr(value)))
self._argv = value
@property
def environ(self):
return getattr(self, '_environ')
@environ.setter
def environ(self, value):
if not (value is None or isinstance(value, dict)):
raise ValueError('Expected a dictionary value for environ, not {}'.format(repr(value)))
self._environ = value
@property
def logger(self):
return self._logger
@property
def path(self):
return self._path
# endregion
# region Methods
def execute(self):
# noinspection PyBroadException
try:
if self._argv is None:
self._argv = os.path.splitext(os.path.basename(self._path))[0]
self._execute(self._path, self._argv, self._environ)
except:
error_type, error, tb = sys.exc_info()
message = 'Command execution failed: ' + six.text_type(error)
self._logger.error(message + '\nTraceback:\n' + ''.join(traceback.format_tb(tb)))
sys.exit(1)
if sys.platform == 'win32':
@staticmethod
def _execute(path, argv=None, environ=None):
""" Executes an external search command.
:param path: Path to the external search command.
:type path: unicode
:param argv: Argument list.
:type argv: list or tuple
The arguments to the child process should start with the name of the command being run, but this is not
enforced. A value of :const:`None` specifies that the base name of path name :param:`path` should be used.
:param environ: A mapping which is used to define the environment variables for the new process.
:type environ: dict or None.
This mapping is used instead of the current process’s environment. A value of :const:`None` specifies that
the :data:`os.environ` mapping should be used.
:return: None
"""
search_path = os.getenv('PATH') if environ is None else environ.get('PATH')
found = ExternalSearchCommand._search_path(path, search_path)
if found is None:
raise ValueError('Cannot find command on path: {}'.format(path))
path = found
logger.debug('starting command="%s", arguments=%s', path, argv)
def terminate(signal_number, frame):
sys.exit('External search command is terminating on receipt of signal={}.'.format(signal_number))
def terminate_child():
if p.pid is not None and p.returncode is None:
logger.debug('terminating command="%s", arguments=%d, pid=%d', path, argv, p.pid)
os.kill(p.pid, CTRL_BREAK_EVENT)
p = Popen(argv, executable=path, env=environ, stdin=sys.stdin, stdout=sys.stdout, stderr=sys.stderr)
atexit.register(terminate_child)
signal(SIGBREAK, terminate)
signal(SIGINT, terminate)
signal(SIGTERM, terminate)
logger.debug('started command="%s", arguments=%s, pid=%d', path, argv, p.pid)
p.wait()
logger.debug('finished command="%s", arguments=%s, pid=%d, returncode=%d', path, argv, p.pid, p.returncode)
if p.returncode != 0:
sys.exit(p.returncode)
@staticmethod
def _search_path(executable, paths):
""" Locates an executable program file.
:param executable: The name of the executable program to locate.
:type executable: unicode
:param paths: A list of one or more directory paths where executable programs are located.
:type paths: unicode
:return:
:rtype: Path to the executable program located or :const:`None`.
"""
directory, filename = os.path.split(executable)
extension = os.path.splitext(filename)[1].upper()
executable_extensions = ExternalSearchCommand._executable_extensions
if directory:
if len(extension) and extension in executable_extensions:
return None
for extension in executable_extensions:
path = executable + extension
if os.path.isfile(path):
return path
return None
if not paths:
return None
directories = [directory for directory in paths.split(';') if len(directory)]
if len(directories) == 0:
return None
if len(extension) and extension in executable_extensions:
for directory in directories:
path = os.path.join(directory, executable)
if os.path.isfile(path):
return path
return None
for directory in directories:
path_without_extension = os.path.join(directory, executable)
for extension in executable_extensions:
path = path_without_extension + extension
if os.path.isfile(path):
return path
return None
_executable_extensions = ('.COM', '.EXE')
else:
@staticmethod
def _execute(path, argv, environ):
if environ is None:
os.execvp(path, argv)
else:
os.execvpe(path, argv, environ)
return
# endregion
def execute(path, argv=None, environ=None, command_class=ExternalSearchCommand):
"""
:param path:
:type path: basestring
:param argv:
:type: argv: list, tuple, or None
:param environ:
:type environ: dict
:param command_class: External search command class to instantiate and execute.
:type command_class: type
:return:
:rtype: None
"""
assert issubclass(command_class, ExternalSearchCommand)
command_class(path, argv, environ).execute()
dist/escu/bin/splunklib/searchcommands/generating_command.py
Vendored
-350
View File
@@ -1,350 +0,0 @@
# coding=utf-8
#
# Copyright © 2011-2015 Splunk, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"): you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from __future__ import absolute_import, division, print_function, unicode_literals
from .decorators import ConfigurationSetting
from .search_command import SearchCommand
from splunklib import six
from splunklib.six.moves import map as imap, filter as ifilter
# P1 [O] TODO: Discuss generates_timeorder in the class-level documentation for GeneratingCommand
class GeneratingCommand(SearchCommand):
""" Generates events based on command arguments.
Generating commands receive no input and must be the first command on a pipeline. There are three pipelines:
streams, events, and reports. The streams pipeline generates or processes time-ordered event records on an
indexer or search head.
Streaming commands filter, modify, or augment event records and can be applied to subsets of index data in a
parallel manner. An example of a streaming command from Splunk's built-in command set is rex_ which extracts and
adds fields to event records at search time. Records that pass through the streams pipeline move on to the events
pipeline.
The events pipeline generates or processes records on a search head. Eventing commands typically filter, group,
order, or augment event records. Examples of eventing commands from Splunk's built-in command set include sort_,
dedup_, and cluster_. Each execution of an eventing command should produce a set of event records that is
independently usable by downstream processors. Records that pass through the events pipeline move on to the reports
pipeline.
The reports pipeline also runs on a search head, but yields data structures for presentation, not event records.
Examples of streaming from Splunk's built-in command set include chart_, stats_, and contingency_.
GeneratingCommand configuration
===============================
Configure your generating command based on the pipeline that it targets. How you configure your command depends on
the Search Command Protocol (SCP) version.
+----------+-------------------------------------+--------------------------------------------+
| Pipeline | SCP 1 | SCP 2 |
+==========+=====================================+============================================+
| streams | streaming=True[,local=[True|False]] | type='streaming'[,distributed=[true|false] |
+----------+-------------------------------------+--------------------------------------------+
| events | retainsevents=True, streaming=False | type='events' |
+----------+-------------------------------------+--------------------------------------------+
| reports | streaming=False | type='reporting' |
+----------+-------------------------------------+--------------------------------------------+
Only streaming commands may be distributed to indexers. By default generating commands are configured to run
locally in the streams pipeline and will run under either SCP 1 or SCP 2.
.. code-block:: python
@Configuration()
class StreamingGeneratingCommand(GeneratingCommand)
...
How you configure your command to run on a different pipeline or in a distributed fashion depends on what SCP
protocol versions you wish to support. You must be sure to configure your command consistently for each protocol,
if you wish to support both protocol versions correctly.
.. _chart: http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Chart
.. _cluster: http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Cluster
.. _contingency: http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Contingency
.. _dedup: http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Dedup
.. _rex: http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Rex
.. _sort: http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Sort
.. _stats: http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Stats
Distributed Generating command
==============================
Commands configured like this will run as the first command on search heads and/or indexers on the streams pipeline.
+----------+---------------------------------------------------+---------------------------------------------------+
| Pipeline | SCP 1 | SCP 2 |
+==========+===================================================+===================================================+
| streams | 1. Add this line to your command's stanza in | 1. Add this configuration setting to your code: |
| | | |
| | default/commands.conf:: | .. code-block:: python |
| | | |
| | local = false | @Configuration(distributed=True) |
| | | class SomeCommand(GeneratingCommand) |
| | | ... |
| | 2. Restart splunk | |
| | | 2. You are good to go; no need to restart Splunk |
+----------+---------------------------------------------------+---------------------------------------------------+
Eventing Generating command
===========================
Generating commands configured like this will run as the first command on a search head on the events pipeline.
+----------+---------------------------------------------------+---------------------------------------------------+
| Pipeline | SCP 1 | SCP 2 |
+==========+===================================================+===================================================+
| events | You have a choice. Add these configuration | Add this configuration setting to your command |
| | settings to your command class: | setting to your command class: |
| | | |
| | .. code-block:: python | .. code-block:: python |
| | | |
| | @Configuration( | @Configuration(type='events') |
| | retainsevents=True, streaming=False) | class SomeCommand(GeneratingCommand) |
| | class SomeCommand(GeneratingCommand) | ... |
| | ... | |
| | | |
| | Or add these lines to default/commands.conf: | |
| | | |
| | .. code-block:: text | |
| | | |
| | retainsevents = true | |
| | streaming = false | |
+----------+---------------------------------------------------+---------------------------------------------------+
Configure your command class like this, if you wish to support both protocols:
.. code-block:: python
@Configuration(type='events', retainsevents=True, streaming=False)
class SomeCommand(GeneratingCommand)
...
You might also consider adding these lines to commands.conf instead of adding them to your command class:
.. code-block:: python
retainsevents = false
streaming = false
Reporting Generating command
============================
Commands configured like this will run as the first command on a search head on the reports pipeline.
+----------+---------------------------------------------------+---------------------------------------------------+
| Pipeline | SCP 1 | SCP 2 |
+==========+===================================================+===================================================+
| events | You have a choice. Add these configuration | Add this configuration setting to your command |
| | settings to your command class: | setting to your command class: |
| | | |
| | .. code-block:: python | .. code-block:: python |
| | | |
| | @Configuration(retainsevents=False) | @Configuration(type='reporting') |
| | class SomeCommand(GeneratingCommand) | class SomeCommand(GeneratingCommand) |
| | ... | ... |
| | | |
| | Or add this lines to default/commands.conf: | |
| | | |
| | .. code-block:: text | |
| | | |
| | retainsevents = false | |
| | streaming = false | |
+----------+---------------------------------------------------+---------------------------------------------------+
Configure your command class like this, if you wish to support both protocols:
.. code-block:: python
@Configuration(type='reporting', streaming=False)
class SomeCommand(GeneratingCommand)
...
You might also consider adding these lines to commands.conf instead of adding them to your command class:
.. code-block:: text
retainsevents = false
streaming = false
"""
# region Methods
def generate(self):
""" A generator that yields records to the Splunk processing pipeline
You must override this method.
"""
raise NotImplementedError('GeneratingCommand.generate(self)')
def _execute(self, ifile, process):
""" Execution loop
:param ifile: Input file object. Unused.
:type ifile: file
:return: `None`.
"""
if self._protocol_version == 2:
self._execute_v2(ifile, self.generate())
else:
assert self._protocol_version == 1
self._record_writer.write_records(self.generate())
self.finish()
def _execute_chunk_v2(self, process, chunk):
count = 0
for row in process:
self._record_writer.write_record(row)
count += 1
if count == self._record_writer._maxresultrows:
self._finished = False
return
self._finished = True
# endregion
# region Types
class ConfigurationSettings(SearchCommand.ConfigurationSettings):
""" Represents the configuration settings for a :code:`GeneratingCommand` class.
"""
# region SCP v1/v2 Properties
generating = ConfigurationSetting(readonly=True, value=True, doc='''
Tells Splunk that this command generates events, but does not process inputs.
Generating commands must appear at the front of the search pipeline identified by :meth:`type`.
Fixed: :const:`True`
Supported by: SCP 1, SCP 2
''')
# endregion
# region SCP v1 Properties
generates_timeorder = ConfigurationSetting(doc='''
:const:`True`, if the command generates new events.
Default: :const:`False`
Supported by: SCP 1
''')
local = ConfigurationSetting(doc='''
:const:`True`, if the command should run locally on the search head.
Default: :const:`False`
Supported by: SCP 1
''')
retainsevents = ConfigurationSetting(doc='''
:const:`True`, if the command retains events the way the sort, dedup, and cluster commands do, or whether it
transforms them the way the stats command does.
Default: :const:`False`
Supported by: SCP 1
''')
streaming = ConfigurationSetting(doc='''
:const:`True`, if the command is streamable.
Default: :const:`True`
Supported by: SCP 1
''')
# endregion
# region SCP v2 Properties
distributed = ConfigurationSetting(value=False, doc='''
True, if this command should be distributed to indexers.
This value is ignored unless :meth:`type` is equal to :const:`streaming`. It is only this command type that
may be distributed.
Default: :const:`False`
Supported by: SCP 2
''')
type = ConfigurationSetting(value='streaming', doc='''
A command type name.
==================== ======================================================================================
Value Description
-------------------- --------------------------------------------------------------------------------------
:const:`'events'` Runs as the first command in the Splunk events pipeline. Cannot be distributed.
:const:`'reporting'` Runs as the first command in the Splunk reports pipeline. Cannot be distributed.
:const:`'streaming'` Runs as the first command in the Splunk streams pipeline. May be distributed.
==================== ======================================================================================
Default: :const:`'streaming'`
Supported by: SCP 2
''')
# endregion
# region Methods
@classmethod
def fix_up(cls, command):
""" Verifies :code:`command` class structure.
"""
if command.generate == GeneratingCommand.generate:
raise AttributeError('No GeneratingCommand.generate override')
# TODO: Stop looking like a dictionary because we don't obey the semantics
# N.B.: Does not use Python 2 dict copy semantics
def iteritems(self):
iteritems = SearchCommand.ConfigurationSettings.iteritems(self)
version = self.command.protocol_version
if version == 2:
iteritems = ifilter(lambda name_value1: name_value1[0] != 'distributed', iteritems)
if not self.distributed and self.type == 'streaming':
iteritems = imap(
lambda name_value: (name_value[0], 'stateful') if name_value[0] == 'type' else (name_value[0], name_value[1]), iteritems)
return iteritems
# N.B.: Does not use Python 3 dict view semantics
if not six.PY2:
items = iteritems
pass
# endregion
pass
# endregion
dist/escu/bin/splunklib/searchcommands/internals.py
Vendored
-844
View File
@@ -1,844 +0,0 @@
# coding=utf-8
#
# Copyright © 2011-2015 Splunk, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"): you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from __future__ import absolute_import, division, print_function
from io import TextIOWrapper
from collections import deque, namedtuple
from splunklib import six
try:
from collections import OrderedDict # must be python 2.7
except ImportError:
from ..ordereddict import OrderedDict
from splunklib.six.moves import StringIO
from itertools import chain
from splunklib.six.moves import map as imap
from json import JSONDecoder, JSONEncoder
from json.encoder import encode_basestring_ascii as json_encode_string
from splunklib.six.moves import urllib
import csv
import gzip
import os
import re
import sys
import warnings
from . import environment
csv.field_size_limit(10485760) # The default value is 128KB; upping to 10MB. See SPL-12117 for background on this issue
def set_binary_mode(fh):
""" Helper method to set up binary mode for file handles.
Emphasis being sys.stdin, sys.stdout, sys.stderr.
For python3, we want to return .buffer
For python2+windows we want to set os.O_BINARY
"""
typefile = TextIOWrapper if sys.version_info >= (3, 0) else file
# check for file handle
if not isinstance(fh, typefile):
return fh
# check for python3 and buffer
if sys.version_info >= (3, 0) and hasattr(fh, 'buffer'):
return fh.buffer
# check for python3
elif sys.version_info >= (3, 0):
pass
# check for windows python2. SPL-175233 -- python3 stdout is already binary
elif sys.platform == 'win32':
# Work around the fact that on Windows '\n' is mapped to '\r\n'. The typical solution is to simply open files in
# binary mode, but stdout is already open, thus this hack. 'CPython' and 'PyPy' work differently. We assume that
# all other Python implementations are compatible with 'CPython'. This might or might not be a valid assumption.
from platform import python_implementation
implementation = python_implementation()
if implementation == 'PyPy':
return os.fdopen(fh.fileno(), 'wb', 0)
else:
import msvcrt
msvcrt.setmode(fh.fileno(), os.O_BINARY)
return fh
class CommandLineParser(object):
r""" Parses the arguments to a search command.
A search command line is described by the following syntax.
**Syntax**::
command = command-name *[wsp option] *[wsp [dquote] field-name [dquote]]
command-name = alpha *( alpha / digit )
option = option-name [wsp] "=" [wsp] option-value
option-name = alpha *( alpha / digit / "_" )
option-value = word / quoted-string
word = 1*( %01-%08 / %0B / %0C / %0E-1F / %21 / %23-%FF ) ; Any character but DQUOTE and WSP
quoted-string = dquote *( word / wsp / "\" dquote / dquote dquote ) dquote
field-name = ( "_" / alpha ) *( alpha / digit / "_" / "." / "-" )
**Note:**
This syntax is constrained to an 8-bit character set.
**Note:**
This syntax does not show that `field-name` values may be comma-separated when in fact they can be. This is
because Splunk strips commas from the command line. A custom search command will never see them.
**Example:**
countmatches fieldname = word_count pattern = \w+ some_text_field
Option names are mapped to properties in the targeted ``SearchCommand``. It is the responsibility of the property
setters to validate the values they receive. Property setters may also produce side effects. For example,
setting the built-in `log_level` immediately changes the `log_level`.
"""
@classmethod
def parse(cls, command, argv):
""" Splits an argument list into an options dictionary and a fieldname
list.
The argument list, `argv`, must be of the form::
*[option]... *[<field-name>]
Options are validated and assigned to items in `command.options`. Field names are validated and stored in the
list of `command.fieldnames`.
#Arguments:
:param command: Search command instance.
:type command: ``SearchCommand``
:param argv: List of search command arguments.
:type argv: ``list``
:return: ``None``
#Exceptions:
``SyntaxError``: Argument list is incorrectly formed.
``ValueError``: Unrecognized option/field name, or an illegal field value.
"""
debug = environment.splunklib_logger.debug
command_class = type(command).__name__
# Prepare
debug('Parsing %s command line: %r', command_class, argv)
command.fieldnames = None
command.options.reset()
argv = ' '.join(argv)
command_args = cls._arguments_re.match(argv)
if command_args is None:
raise SyntaxError('Syntax error: {}'.format(argv))
# Parse options
for option in cls._options_re.finditer(command_args.group('options')):
name, value = option.group('name'), option.group('value')
if name not in command.options:
raise ValueError(
'Unrecognized {} command option: {}={}'.format(command.name, name, json_encode_string(value)))
command.options[name].value = cls.unquote(value)
missing = command.options.get_missing()
if missing is not None:
if len(missing) > 1:
raise ValueError(
'Values for these {} command options are required: {}'.format(command.name, ', '.join(missing)))
raise ValueError('A value for {} command option {} is required'.format(command.name, missing[0]))
# Parse field names
fieldnames = command_args.group('fieldnames')
if fieldnames is None:
command.fieldnames = []
else:
command.fieldnames = [cls.unquote(value.group(0)) for value in cls._fieldnames_re.finditer(fieldnames)]
debug(' %s: %s', command_class, command)
@classmethod
def unquote(cls, string):
""" Removes quotes from a quoted string.
Splunk search command quote rules are applied. The enclosing double-quotes, if present, are removed. Escaped
double-quotes ('\"' or '""') are replaced by a single double-quote ('"').
**NOTE**
We are not using a json.JSONDecoder because Splunk quote rules are different than JSON quote rules. A
json.JSONDecoder does not recognize a pair of double-quotes ('""') as an escaped quote ('"') and will
decode single-quoted strings ("'") in addition to double-quoted ('"') strings.
"""
if len(string) == 0:
return ''
if string[0] == '"':
if len(string) == 1 or string[-1] != '"':
raise SyntaxError('Poorly formed string literal: ' + string)
string = string[1:-1]
if len(string) == 0:
return ''
def replace(match):
value = match.group(0)
if value == '""':
return '"'
if len(value) < 2:
raise SyntaxError('Poorly formed string literal: ' + string)
return value[1]
result = re.sub(cls._escaped_character_re, replace, string)
return result
# region Class variables
_arguments_re = re.compile(r"""
^\s*
(?P<options> # Match a leading set of name/value pairs
(?:
(?:(?=\w)[^\d]\w*) # name
\s*=\s* # =
(?:"(?:\\.|""|[^"])*"|(?:\\.|[^\s"])+)\s* # value
)*
)\s*
(?P<fieldnames> # Match a trailing set of field names
(?:
(?:"(?:\\.|""|[^"])*"|(?:\\.|[^\s"])+)\s*
)*
)\s*$
""", re.VERBOSE | re.UNICODE)
_escaped_character_re = re.compile(r'(\\.|""|[\\"])')
_fieldnames_re = re.compile(r"""("(?:\\.|""|[^"\\])+"|(?:\\.|[^\s"])+)""")
_options_re = re.compile(r"""
# Captures a set of name/value pairs when used with re.finditer
(?P<name>(?:(?=\w)[^\d]\w*)) # name
\s*=\s* # =
(?P<value>"(?:\\.|""|[^"])*"|(?:\\.|[^\s"])+) # value
""", re.VERBOSE | re.UNICODE)
# endregion
class ConfigurationSettingsType(type):
""" Metaclass for constructing ConfigurationSettings classes.
Instances of :class:`ConfigurationSettingsType` construct :class:`ConfigurationSettings` classes from classes from
a base :class:`ConfigurationSettings` class and a dictionary of configuration settings. The settings in the
dictionary are validated against the settings in the base class. You cannot add settings, you can only change their
backing-field values and you cannot modify settings without backing-field values. These are considered fixed
configuration setting values.
This is an internal class used in two places:
+ :meth:`decorators.Configuration.__call__`
Adds a ConfigurationSettings attribute to a :class:`SearchCommand` class.
+ :meth:`reporting_command.ReportingCommand.fix_up`
Adds a ConfigurationSettings attribute to a :meth:`ReportingCommand.map` method, if there is one.
"""
def __new__(mcs, module, name, bases):
mcs = super(ConfigurationSettingsType, mcs).__new__(mcs, str(name), bases, {})
return mcs
def __init__(cls, module, name, bases):
super(ConfigurationSettingsType, cls).__init__(name, bases, None)
cls.__module__ = module
@staticmethod
def validate_configuration_setting(specification, name, value):
if not isinstance(value, specification.type):
if isinstance(specification.type, type):
type_names = specification.type.__name__
else:
type_names = ', '.join(imap(lambda t: t.__name__, specification.type))
raise ValueError('Expected {} value, not {}={}'.format(type_names, name, repr(value)))
if specification.constraint and not specification.constraint(value):
raise ValueError('Illegal value: {}={}'.format(name, repr(value)))
return value
specification = namedtuple(
'ConfigurationSettingSpecification', (
'type',
'constraint',
'supporting_protocols'))
# P1 [ ] TODO: Review ConfigurationSettingsType.specification_matrix for completeness and correctness
specification_matrix = {
'clear_required_fields': specification(
type=bool,
constraint=None,
supporting_protocols=[1]),
'distributed': specification(
type=bool,
constraint=None,
supporting_protocols=[2]),
'generates_timeorder': specification(
type=bool,
constraint=None,
supporting_protocols=[1]),
'generating': specification(
type=bool,
constraint=None,
supporting_protocols=[1, 2]),
'local': specification(
type=bool,
constraint=None,
supporting_protocols=[1]),
'maxinputs': specification(
type=int,
constraint=lambda value: 0 <= value <= six.MAXSIZE,
supporting_protocols=[2]),
'overrides_timeorder': specification(
type=bool,
constraint=None,
supporting_protocols=[1]),
'required_fields': specification(
type=(list, set, tuple),
constraint=None,
supporting_protocols=[1, 2]),
'requires_preop': specification(
type=bool,
constraint=None,
supporting_protocols=[1]),
'retainsevents': specification(
type=bool,
constraint=None,
supporting_protocols=[1]),
'run_in_preview': specification(
type=bool,
constraint=None,
supporting_protocols=[2]),
'streaming': specification(
type=bool,
constraint=None,
supporting_protocols=[1]),
'streaming_preop': specification(
type=(bytes, six.text_type),
constraint=None,
supporting_protocols=[1, 2]),
'type': specification(
type=(bytes, six.text_type),
constraint=lambda value: value in ('events', 'reporting', 'streaming'),
supporting_protocols=[2])}
class CsvDialect(csv.Dialect):
""" Describes the properties of Splunk CSV streams """
delimiter = ','
quotechar = '"'
doublequote = True
skipinitialspace = False
lineterminator = '\r\n'
if sys.version_info >= (3, 0) and sys.platform == 'win32':
lineterminator = '\n'
quoting = csv.QUOTE_MINIMAL
class InputHeader(dict):
""" Represents a Splunk input header as a collection of name/value pairs.
"""
def __str__(self):
return '\n'.join([name + ':' + value for name, value in six.iteritems(self)])
def read(self, ifile):
""" Reads an input header from an input file.
The input header is read as a sequence of *<name>***:***<value>* pairs separated by a newline. The end of the
input header is signalled by an empty line or an end-of-file.
:param ifile: File-like object that supports iteration over lines.
"""
name, value = None, None
for line in ifile:
if line == '\n':
break
item = line.split(':', 1)
if len(item) == 2:
# start of a new item
if name is not None:
self[name] = value[:-1] # value sans trailing newline
name, value = item[0], urllib.parse.unquote(item[1])
elif name is not None:
# continuation of the current item
value += urllib.parse.unquote(line)
if name is not None:
self[name] = value[:-1] if value[-1] == '\n' else value
Message = namedtuple('Message', ('type', 'text'))
class MetadataDecoder(JSONDecoder):
def __init__(self):
JSONDecoder.__init__(self, object_hook=self._object_hook)
@staticmethod
def _object_hook(dictionary):
object_view = ObjectView(dictionary)
stack = deque()
stack.append((None, None, dictionary))
while len(stack):
instance, member_name, dictionary = stack.popleft()
for name, value in six.iteritems(dictionary):
if isinstance(value, dict):
stack.append((dictionary, name, value))
if instance is not None:
instance[member_name] = ObjectView(dictionary)
return object_view
class MetadataEncoder(JSONEncoder):
def __init__(self):
JSONEncoder.__init__(self, separators=MetadataEncoder._separators)
def default(self, o):
return o.__dict__ if isinstance(o, ObjectView) else JSONEncoder.default(self, o)
_separators = (',', ':')
class ObjectView(object):
def __init__(self, dictionary):
self.__dict__ = dictionary
def __repr__(self):
return repr(self.__dict__)
def __str__(self):
return str(self.__dict__)
class Recorder(object):
def __init__(self, path, f):
self._recording = gzip.open(path + '.gz', 'wb')
self._file = f
def __getattr__(self, name):
return getattr(self._file, name)
def __iter__(self):
for line in self._file:
self._recording.write(line)
self._recording.flush()
yield line
def read(self, size=None):
value = self._file.read() if size is None else self._file.read(size)
self._recording.write(value)
self._recording.flush()
return value
def readline(self, size=None):
value = self._file.readline() if size is None else self._file.readline(size)
if len(value) > 0:
self._recording.write(value)
self._recording.flush()
return value
def record(self, *args):
for arg in args:
self._recording.write(arg)
def write(self, text):
self._recording.write(text)
self._file.write(text)
self._recording.flush()
class RecordWriter(object):
def __init__(self, ofile, maxresultrows=None):
self._maxresultrows = 50000 if maxresultrows is None else maxresultrows
self._ofile = set_binary_mode(ofile)
self._fieldnames = None
self._buffer = StringIO()
self._writer = csv.writer(self._buffer, dialect=CsvDialect)
self._writerow = self._writer.writerow
self._finished = False
self._flushed = False
self._inspector = OrderedDict()
self._chunk_count = 0
self._pending_record_count = 0
self._committed_record_count = 0
@property
def is_flushed(self):
return self._flushed
@is_flushed.setter
def is_flushed(self, value):
self._flushed = True if value else False
@property
def ofile(self):
return self._ofile
@ofile.setter
def ofile(self, value):
self._ofile = set_binary_mode(value)
@property
def pending_record_count(self):
return self._pending_record_count
@property
def _record_count(self):
warnings.warn(
"_record_count will be deprecated soon. Use pending_record_count instead.",
PendingDeprecationWarning
)
return self.pending_record_count
@property
def committed_record_count(self):
return self._committed_record_count
@property
def _total_record_count(self):
warnings.warn(
"_total_record_count will be deprecated soon. Use committed_record_count instead.",
PendingDeprecationWarning
)
return self.committed_record_count
def write(self, data):
bytes_type = bytes if sys.version_info >= (3, 0) else str
if not isinstance(data, bytes_type):
data = data.encode('utf-8')
self.ofile.write(data)
def flush(self, finished=None, partial=None):
assert finished is None or isinstance(finished, bool)
assert partial is None or isinstance(partial, bool)
assert not (finished is None and partial is None)
assert finished is None or partial is None
self._ensure_validity()
def write_message(self, message_type, message_text, *args, **kwargs):
self._ensure_validity()
self._inspector.setdefault('messages', []).append((message_type, message_text.format(*args, **kwargs)))
def write_record(self, record):
self._ensure_validity()
self._write_record(record)
def write_records(self, records):
self._ensure_validity()
write_record = self._write_record
for record in records:
write_record(record)
def _clear(self):
self._buffer.seek(0)
self._buffer.truncate()
self._inspector.clear()
self._pending_record_count = 0
def _ensure_validity(self):
if self._finished is True:
assert self._record_count == 0 and len(self._inspector) == 0
raise RuntimeError('I/O operation on closed record writer')
def _write_record(self, record):
fieldnames = self._fieldnames
if fieldnames is None:
self._fieldnames = fieldnames = list(record.keys())
value_list = imap(lambda fn: (str(fn), str('__mv_') + str(fn)), fieldnames)
self._writerow(list(chain.from_iterable(value_list)))
get_value = record.get
values = []
for fieldname in fieldnames:
value = get_value(fieldname, None)
if value is None:
values += (None, None)
continue
value_t = type(value)
if issubclass(value_t, (list, tuple)):
if len(value) == 0:
values += (None, None)
continue
if len(value) > 1:
value_list = value
sv = ''
mv = '$'
for value in value_list:
if value is None:
sv += '\n'
mv += '$;$'
continue
value_t = type(value)
if value_t is not bytes:
if value_t is bool:
value = str(value.real)
elif value_t is six.text_type:
value = value
elif isinstance(value, six.integer_types) or value_t is float or value_t is complex:
value = str(value)
elif issubclass(value_t, (dict, list, tuple)):
value = str(''.join(RecordWriter._iterencode_json(value, 0)))
else:
value = repr(value).encode('utf-8', errors='backslashreplace')
sv += value + '\n'
mv += value.replace('$', '$$') + '$;$'
values += (sv[:-1], mv[:-2])
continue
value = value[0]
value_t = type(value)
if value_t is bool:
values += (str(value.real), None)
continue
if value_t is bytes:
values += (value, None)
continue
if value_t is six.text_type:
if six.PY2:
value = value.encode('utf-8')
values += (value, None)
continue
if isinstance(value, six.integer_types) or value_t is float or value_t is complex:
values += (str(value), None)
continue
if issubclass(value_t, dict):
values += (str(''.join(RecordWriter._iterencode_json(value, 0))), None)
continue
values += (repr(value), None)
self._writerow(values)
self._pending_record_count += 1
if self.pending_record_count >= self._maxresultrows:
self.flush(partial=True)
try:
# noinspection PyUnresolvedReferences
from _json import make_encoder
except ImportError:
# We may be running under PyPy 2.5 which does not include the _json module
_iterencode_json = JSONEncoder(separators=(',', ':')).iterencode
else:
# Creating _iterencode_json this way yields a two-fold performance improvement on Python 2.7.9 and 2.7.10
from json.encoder import encode_basestring_ascii
@staticmethod
def _default(o):
raise TypeError(repr(o) + ' is not JSON serializable')
_iterencode_json = make_encoder(
{}, # markers (for detecting circular references)
_default, # object_encoder
encode_basestring_ascii, # string_encoder
None, # indent
':', ',', # separators
False, # sort_keys
False, # skip_keys
True # allow_nan
)
del make_encoder
class RecordWriterV1(RecordWriter):
def flush(self, finished=None, partial=None):
RecordWriter.flush(self, finished, partial) # validates arguments and the state of this instance
if self.pending_record_count > 0 or (self._chunk_count == 0 and 'messages' in self._inspector):
messages = self._inspector.get('messages')
if self._chunk_count == 0:
# Messages are written to the messages header when we write the first chunk of data
# Guarantee: These messages are displayed by splunkweb and the job inspector
if messages is not None:
message_level = RecordWriterV1._message_level.get
for level, text in messages:
self.write(message_level(level, level))
self.write('=')
self.write(text)
self.write('\r\n')
self.write('\r\n')
elif messages is not None:
# Messages are written to the messages header when we write subsequent chunks of data
# Guarantee: These messages are displayed by splunkweb and the job inspector, if and only if the
# command is configured with
#
# stderr_dest = message
#
# stderr_dest is a static configuration setting. This means that it can only be set in commands.conf.
# It cannot be set in code.
stderr = sys.stderr
for level, text in messages:
print(level, text, file=stderr)
self.write(self._buffer.getvalue())
self._chunk_count += 1
self._committed_record_count += self.pending_record_count
self._clear()
self._finished = finished is True
_message_level = {
'DEBUG': 'debug_message',
'ERROR': 'error_message',
'FATAL': 'error_message',
'INFO': 'info_message',
'WARN': 'warn_message'
}
class RecordWriterV2(RecordWriter):
def flush(self, finished=None, partial=None):
RecordWriter.flush(self, finished, partial) # validates arguments and the state of this instance
if partial or not finished:
# Don't flush partial chunks, since the SCP v2 protocol does not
# provide a way to send partial chunks yet.
return
if not self.is_flushed:
self.write_chunk(finished=True)
def write_chunk(self, finished=None):
inspector = self._inspector
self._committed_record_count += self.pending_record_count
self._chunk_count += 1
# TODO: DVPL-6448: splunklib.searchcommands | Add support for partial: true when it is implemented in
# ChunkedExternProcessor (See SPL-103525)
#
# We will need to replace the following block of code with this block:
#
# metadata = [item for item in (('inspector', inspector), ('finished', finished), ('partial', partial))]
#
# if partial is True:
# finished = False
if len(inspector) == 0:
inspector = None
metadata = [item for item in (('inspector', inspector), ('finished', finished))]
self._write_chunk(metadata, self._buffer.getvalue())
self._clear()
def write_metadata(self, configuration):
self._ensure_validity()
metadata = chain(six.iteritems(configuration), (('inspector', self._inspector if self._inspector else None),))
self._write_chunk(metadata, '')
self.write('\n')
self._clear()
def write_metric(self, name, value):
self._ensure_validity()
self._inspector['metric.' + name] = value
def _clear(self):
super(RecordWriterV2, self)._clear()
self._fieldnames = None
def _write_chunk(self, metadata, body):
if metadata:
metadata = str(''.join(self._iterencode_json(dict([(n, v) for n, v in metadata if v is not None]), 0)))
if sys.version_info >= (3, 0):
metadata = metadata.encode('utf-8')
metadata_length = len(metadata)
else:
metadata_length = 0
if sys.version_info >= (3, 0):
body = body.encode('utf-8')
body_length = len(body)
if not (metadata_length > 0 or body_length > 0):
return
start_line = 'chunked 1.0,%s,%s\n' % (metadata_length, body_length)
self.write(start_line)
self.write(metadata)
self.write(body)
self._ofile.flush()
self._flushed = True
dist/escu/bin/splunklib/searchcommands/reporting_command.py
Vendored
-281
View File
@@ -1,281 +0,0 @@
# coding=utf-8
#
# Copyright © 2011-2015 Splunk, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"): you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from __future__ import absolute_import, division, print_function, unicode_literals
from itertools import chain
from .internals import ConfigurationSettingsType, json_encode_string
from .decorators import ConfigurationSetting, Option
from .streaming_command import StreamingCommand
from .search_command import SearchCommand
from .validators import Set
from splunklib import six
class ReportingCommand(SearchCommand):
""" Processes search result records and generates a reporting data structure.
Reporting search commands run as either reduce or map/reduce operations. The reduce part runs on a search head and
is responsible for processing a single chunk of search results to produce the command's reporting data structure.
The map part is called a streaming preop. It feeds the reduce part with partial results and by default runs on the
search head and/or one or more indexers.
You must implement a :meth:`reduce` method as a generator function that iterates over a set of event records and
yields a reporting data structure. You may implement a :meth:`map` method as a generator function that iterates
over a set of event records and yields :class:`dict` or :class:`list(dict)` instances.
ReportingCommand configuration
==============================
Configure the :meth:`map` operation using a Configuration decorator on your :meth:`map` method. Configure it like
you would a :class:`StreamingCommand`. Configure the :meth:`reduce` operation using a Configuration decorator on
your :meth:`ReportingCommand` class.
You can configure your command for operation under Search Command Protocol (SCP) version 1 or 2. SCP 2 requires
Splunk 6.3 or later.
"""
# region Special methods
def __init__(self):
SearchCommand.__init__(self)
# endregion
# region Options
phase = Option(doc='''
**Syntax:** phase=[map|reduce]
**Description:** Identifies the phase of the current map-reduce operation.
''', default='reduce', validate=Set('map', 'reduce'))
# endregion
# region Methods
def map(self, records):
""" Override this method to compute partial results.
:param records:
:type records:
You must override this method, if :code:`requires_preop=True`.
"""
return NotImplemented
def prepare(self):
phase = self.phase
if phase == 'map':
# noinspection PyUnresolvedReferences
self._configuration = self.map.ConfigurationSettings(self)
return
if phase == 'reduce':
streaming_preop = chain((self.name, 'phase="map"', str(self._options)), self.fieldnames)
self._configuration.streaming_preop = ' '.join(streaming_preop)
return
raise RuntimeError('Unrecognized reporting command phase: {}'.format(json_encode_string(six.text_type(phase))))
def reduce(self, records):
""" Override this method to produce a reporting data structure.
You must override this method.
"""
raise NotImplementedError('reduce(self, records)')
def _execute(self, ifile, process):
SearchCommand._execute(self, ifile, getattr(self, self.phase))
# endregion
# region Types
class ConfigurationSettings(SearchCommand.ConfigurationSettings):
""" Represents the configuration settings for a :code:`ReportingCommand`.
"""
# region SCP v1/v2 Properties
required_fields = ConfigurationSetting(doc='''
List of required fields for this search which back-propagates to the generating search.
Setting this value enables selected fields mode under SCP 2. Under SCP 1 you must also specify
:code:`clear_required_fields=True` to enable selected fields mode. To explicitly select all fields,
specify a value of :const:`['*']`. No error is generated if a specified field is missing.
Default: :const:`None`, which implicitly selects all fields.
Supported by: SCP 1, SCP 2
''')
requires_preop = ConfigurationSetting(doc='''
Indicates whether :meth:`ReportingCommand.map` is required for proper command execution.
If :const:`True`, :meth:`ReportingCommand.map` is guaranteed to be called. If :const:`False`, Splunk
considers it to be an optimization that may be skipped.
Default: :const:`False`
Supported by: SCP 1, SCP 2
''')
streaming_preop = ConfigurationSetting(doc='''
Denotes the requested streaming preop search string.
Computed.
Supported by: SCP 1, SCP 2
''')
# endregion
# region SCP v1 Properties
clear_required_fields = ConfigurationSetting(doc='''
:const:`True`, if required_fields represent the *only* fields required.
If :const:`False`, required_fields are additive to any fields that may be required by subsequent commands.
In most cases, :const:`True` is appropriate for reporting commands.
Default: :const:`True`
Supported by: SCP 1
''')
retainsevents = ConfigurationSetting(readonly=True, value=False, doc='''
Signals that :meth:`ReportingCommand.reduce` transforms _raw events to produce a reporting data structure.
Fixed: :const:`False`
Supported by: SCP 1
''')
streaming = ConfigurationSetting(readonly=True, value=False, doc='''
Signals that :meth:`ReportingCommand.reduce` runs on the search head.
Fixed: :const:`False`
Supported by: SCP 1
''')
# endregion
# region SCP v2 Properties
maxinputs = ConfigurationSetting(doc='''
Specifies the maximum number of events that can be passed to the command for each invocation.
This limit cannot exceed the value of `maxresultrows` in limits.conf_. Under SCP 1 you must specify this
value in commands.conf_.
Default: The value of `maxresultrows`.
Supported by: SCP 2
.. _limits.conf: http://docs.splunk.com/Documentation/Splunk/latest/admin/Limitsconf
''')
run_in_preview = ConfigurationSetting(doc='''
:const:`True`, if this command should be run to generate results for preview; not wait for final output.
This may be important for commands that have side effects (e.g., outputlookup).
Default: :const:`True`
Supported by: SCP 2
''')
type = ConfigurationSetting(readonly=True, value='reporting', doc='''
Command type name.
Fixed: :const:`'reporting'`.
Supported by: SCP 2
''')
# endregion
# region Methods
@classmethod
def fix_up(cls, command):
""" Verifies :code:`command` class structure and configures the :code:`command.map` method.
Verifies that :code:`command` derives from :class:`ReportingCommand` and overrides
:code:`ReportingCommand.reduce`. It then configures :code:`command.reduce`, if an overriding implementation
of :code:`ReportingCommand.reduce` has been provided.
:param command: :code:`ReportingCommand` class
Exceptions:
:code:`TypeError` :code:`command` class is not derived from :code:`ReportingCommand`
:code:`AttributeError` No :code:`ReportingCommand.reduce` override
"""
if not issubclass(command, ReportingCommand):
raise TypeError('{} is not a ReportingCommand'.format( command))
if command.reduce == ReportingCommand.reduce:
raise AttributeError('No ReportingCommand.reduce override')
if command.map == ReportingCommand.map:
cls._requires_preop = False
return
f = vars(command)['map'] # Function backing the map method
# EXPLANATION OF PREVIOUS STATEMENT: There is no way to add custom attributes to methods. See [Why does
# setattr fail on a method](http://stackoverflow.com/questions/7891277/why-does-setattr-fail-on-a-bound-method) for a discussion of this issue.
try:
settings = f._settings
except AttributeError:
f.ConfigurationSettings = StreamingCommand.ConfigurationSettings
return
# Create new StreamingCommand.ConfigurationSettings class
module = command.__module__ + '.' + command.__name__ + '.map'
name = b'ConfigurationSettings'
bases = (StreamingCommand.ConfigurationSettings,)
f.ConfigurationSettings = ConfigurationSettingsType(module, name, bases)
ConfigurationSetting.fix_up(f.ConfigurationSettings, settings)
del f._settings
pass
# endregion
pass
# endregion
dist/escu/bin/splunklib/searchcommands/search_command.py
Vendored
-1127
View File
File diff suppressed because it is too large Load Diff
dist/escu/bin/splunklib/searchcommands/streaming_command.py
Vendored
-195
View File
@@ -1,195 +0,0 @@
# coding=utf-8
#
# Copyright 2011-2015 Splunk, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"): you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from __future__ import absolute_import, division, print_function, unicode_literals
from splunklib import six
from splunklib.six.moves import map as imap, filter as ifilter
from .decorators import ConfigurationSetting
from .search_command import SearchCommand
class StreamingCommand(SearchCommand):
""" Applies a transformation to search results as they travel through the streams pipeline.
Streaming commands typically filter, augment, or update, search result records. Splunk will send them in batches of
up to 50,000 records. Hence, a search command must be prepared to be invoked many times during the course of
pipeline processing. Each invocation should produce a set of results independently usable by downstream processors.
By default Splunk may choose to run a streaming command locally on a search head and/or remotely on one or more
indexers concurrently. The size and frequency of the search result batches sent to the command will vary based
on scheduling considerations.
StreamingCommand configuration
==============================
You can configure your command for operation under Search Command Protocol (SCP) version 1 or 2. SCP 2 requires
Splunk 6.3 or later.
"""
# region Methods
def stream(self, records):
""" Generator function that processes and yields event records to the Splunk stream pipeline.
You must override this method.
"""
raise NotImplementedError('StreamingCommand.stream(self, records)')
def _execute(self, ifile, process):
SearchCommand._execute(self, ifile, self.stream)
# endregion
class ConfigurationSettings(SearchCommand.ConfigurationSettings):
""" Represents the configuration settings that apply to a :class:`StreamingCommand`.
"""
# region SCP v1/v2 properties
required_fields = ConfigurationSetting(doc='''
List of required fields for this search which back-propagates to the generating search.
Setting this value enables selected fields mode under SCP 2. Under SCP 1 you must also specify
:code:`clear_required_fields=True` to enable selected fields mode. To explicitly select all fields,
specify a value of :const:`['*']`. No error is generated if a specified field is missing.
Default: :const:`None`, which implicitly selects all fields.
Supported by: SCP 1, SCP 2
''')
# endregion
# region SCP v1 properties
clear_required_fields = ConfigurationSetting(doc='''
:const:`True`, if required_fields represent the *only* fields required.
If :const:`False`, required_fields are additive to any fields that may be required by subsequent commands.
In most cases, :const:`False` is appropriate for streaming commands.
Default: :const:`False`
Supported by: SCP 1
''')
local = ConfigurationSetting(doc='''
:const:`True`, if the command should run locally on the search head.
Default: :const:`False`
Supported by: SCP 1
''')
overrides_timeorder = ConfigurationSetting(doc='''
:const:`True`, if the command changes the order of events with respect to time.
Default: :const:`False`
Supported by: SCP 1
''')
streaming = ConfigurationSetting(readonly=True, value=True, doc='''
Specifies that the command is streamable.
Fixed: :const:`True`
Supported by: SCP 1
''')
# endregion
# region SCP v2 Properties
distributed = ConfigurationSetting(value=True, doc='''
:const:`True`, if this command should be distributed to indexers.
Under SCP 1 you must either specify `local = False` or include this line in commands.conf_, if this command
should be distributed to indexers.
..code:
local = true
Default: :const:`True`
Supported by: SCP 2
.. commands.conf_: http://docs.splunk.com/Documentation/Splunk/latest/Admin/Commandsconf
''')
maxinputs = ConfigurationSetting(doc='''
Specifies the maximum number of events that can be passed to the command for each invocation.
This limit cannot exceed the value of `maxresultrows` in limits.conf. Under SCP 1 you must specify this
value in commands.conf_.
Default: The value of `maxresultrows`.
Supported by: SCP 2
''')
type = ConfigurationSetting(readonly=True, value='streaming', doc='''
Command type name.
Fixed: :const:`'streaming'`
Supported by: SCP 2
''')
# endregion
# region Methods
@classmethod
def fix_up(cls, command):
""" Verifies :code:`command` class structure.
"""
if command.stream == StreamingCommand.stream:
raise AttributeError('No StreamingCommand.stream override')
return
# TODO: Stop looking like a dictionary because we don't obey the semantics
# N.B.: Does not use Python 2 dict copy semantics
def iteritems(self):
iteritems = SearchCommand.ConfigurationSettings.iteritems(self)
version = self.command.protocol_version
if version == 1:
if self.required_fields is None:
iteritems = ifilter(lambda name_value: name_value[0] != 'clear_required_fields', iteritems)
else:
iteritems = ifilter(lambda name_value2: name_value2[0] != 'distributed', iteritems)
if not self.distributed:
iteritems = imap(
lambda name_value1: (name_value1[0], 'stateful') if name_value1[0] == 'type' else (name_value1[0], name_value1[1]), iteritems)
return iteritems
# N.B.: Does not use Python 3 dict view semantics
if not six.PY2:
items = iteritems
# endregion
dist/escu/bin/splunklib/searchcommands/validators.py
Vendored
-394
View File
@@ -1,394 +0,0 @@
# coding=utf-8
#
# Copyright 2011-2015 Splunk, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"): you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from __future__ import absolute_import, division, print_function, unicode_literals
from json.encoder import encode_basestring_ascii as json_encode_string
from collections import namedtuple
from splunklib.six.moves import StringIO
from io import open
import csv
import os
import re
from splunklib import six
from splunklib.six.moves import getcwd
class Validator(object):
""" Base class for validators that check and format search command options.
You must inherit from this class and override :code:`Validator.__call__` and
:code:`Validator.format`. :code:`Validator.__call__` should convert the
value it receives as argument and then return it or raise a
:code:`ValueError`, if the value will not convert.
:code:`Validator.format` should return a human readable version of the value
it receives as argument the same way :code:`str` does.
"""
def __call__(self, value):
raise NotImplementedError()
def format(self, value):
raise NotImplementedError()
class Boolean(Validator):
""" Validates Boolean option values.
"""
truth_values = {
'1': True, '0': False,
't': True, 'f': False,
'true': True, 'false': False,
'y': True, 'n': False,
'yes': True, 'no': False
}
def __call__(self, value):
if not (value is None or isinstance(value, bool)):
value = six.text_type(value).lower()
if value not in Boolean.truth_values:
raise ValueError('Unrecognized truth value: {0}'.format(value))
value = Boolean.truth_values[value]
return value
def format(self, value):
return None if value is None else 't' if value else 'f'
class Code(Validator):
""" Validates code option values.
This validator compiles an option value into a Python code object that can be executed by :func:`exec` or evaluated
by :func:`eval`. The value returned is a :func:`namedtuple` with two members: object, the result of compilation, and
source, the original option value.
"""
def __init__(self, mode='eval'):
"""
:param mode: Specifies what kind of code must be compiled; it can be :const:`'exec'`, if source consists of a
sequence of statements, :const:`'eval'`, if it consists of a single expression, or :const:`'single'` if it
consists of a single interactive statement. In the latter case, expression statements that evaluate to
something other than :const:`None` will be printed.
:type mode: unicode or bytes
"""
self._mode = mode
def __call__(self, value):
if value is None:
return None
try:
return Code.object(compile(value, 'string', self._mode), six.text_type(value))
except (SyntaxError, TypeError) as error:
if six.PY2:
message = error.message
else:
message = str(error)
six.raise_from(ValueError(message), error)
def format(self, value):
return None if value is None else value.source
object = namedtuple('Code', ('object', 'source'))
class Fieldname(Validator):
""" Validates field name option values.
"""
pattern = re.compile(r'''[_.a-zA-Z-][_.a-zA-Z0-9-]*$''')
def __call__(self, value):
if value is not None:
value = six.text_type(value)
if Fieldname.pattern.match(value) is None:
raise ValueError('Illegal characters in fieldname: {}'.format(value))
return value
def format(self, value):
return value
class File(Validator):
""" Validates file option values.
"""
def __init__(self, mode='rt', buffering=None, directory=None):
self.mode = mode
self.buffering = buffering
self.directory = File._var_run_splunk if directory is None else directory
def __call__(self, value):
if value is None:
return value
path = six.text_type(value)
if not os.path.isabs(path):
path = os.path.join(self.directory, path)
try:
value = open(path, self.mode) if self.buffering is None else open(path, self.mode, self.buffering)
except IOError as error:
raise ValueError('Cannot open {0} with mode={1} and buffering={2}: {3}'.format(
value, self.mode, self.buffering, error))
return value
def format(self, value):
return None if value is None else value.name
_var_run_splunk = os.path.join(
os.environ['SPLUNK_HOME'] if 'SPLUNK_HOME' in os.environ else getcwd(), 'var', 'run', 'splunk')
class Integer(Validator):
""" Validates integer option values.
"""
def __init__(self, minimum=None, maximum=None):
if minimum is not None and maximum is not None:
def check_range(value):
if not (minimum <= value <= maximum):
raise ValueError('Expected integer in the range [{0},{1}], not {2}'.format(minimum, maximum, value))
return
elif minimum is not None:
def check_range(value):
if value < minimum:
raise ValueError('Expected integer in the range [{0},+∞], not {1}'.format(minimum, value))
return
elif maximum is not None:
def check_range(value):
if value > maximum:
raise ValueError('Expected integer in the range [-∞,{0}], not {1}'.format(maximum, value))
return
else:
def check_range(value):
return
self.check_range = check_range
return
def __call__(self, value):
if value is None:
return None
try:
if six.PY2:
value = long(value)
else:
value = int(value)
except ValueError:
raise ValueError('Expected integer value, not {}'.format(json_encode_string(value)))
self.check_range(value)
return value
def format(self, value):
return None if value is None else six.text_type(int(value))
class Duration(Validator):
""" Validates duration option values.
"""
def __call__(self, value):
if value is None:
return None
p = value.split(':', 2)
result = None
_60 = Duration._60
_unsigned = Duration._unsigned
try:
if len(p) == 1:
result = _unsigned(p[0])
if len(p) == 2:
result = 60 * _unsigned(p[0]) + _60(p[1])
if len(p) == 3:
result = 3600 * _unsigned(p[0]) + 60 * _60(p[1]) + _60(p[2])
except ValueError:
raise ValueError('Invalid duration value: {0}'.format(value))
return result
def format(self, value):
if value is None:
return None
value = int(value)
s = value % 60
m = value // 60 % 60
h = value // (60 * 60)
return '{0:02d}:{1:02d}:{2:02d}'.format(h, m, s)
_60 = Integer(0, 59)
_unsigned = Integer(0)
class List(Validator):
""" Validates a list of strings
"""
class Dialect(csv.Dialect):
""" Describes the properties of list option values. """
strict = True
delimiter = str(',')
quotechar = str('"')
doublequote = True
lineterminator = str('\n')
skipinitialspace = True
quoting = csv.QUOTE_MINIMAL
def __init__(self, validator=None):
if not (validator is None or isinstance(validator, Validator)):
raise ValueError('Expected a Validator instance or None for validator, not {}', repr(validator))
self._validator = validator
def __call__(self, value):
if value is None or isinstance(value, list):
return value
try:
value = next(csv.reader([value], self.Dialect))
except csv.Error as error:
raise ValueError(error)
if self._validator is None:
return value
try:
for index, item in enumerate(value):
value[index] = self._validator(item)
except ValueError as error:
raise ValueError('Could not convert item {}: {}'.format(index, error))
return value
def format(self, value):
output = StringIO()
writer = csv.writer(output, List.Dialect)
writer.writerow(value)
value = output.getvalue()
return value[:-1]
class Map(Validator):
""" Validates map option values.
"""
def __init__(self, **kwargs):
self.membership = kwargs
def __call__(self, value):
if value is None:
return None
value = six.text_type(value)
if value not in self.membership:
raise ValueError('Unrecognized value: {0}'.format(value))
return self.membership[value]
def format(self, value):
return None if value is None else list(self.membership.keys())[list(self.membership.values()).index(value)]
class Match(Validator):
""" Validates that a value matches a regular expression pattern.
"""
def __init__(self, name, pattern, flags=0):
self.name = six.text_type(name)
self.pattern = re.compile(pattern, flags)
def __call__(self, value):
if value is None:
return None
value = six.text_type(value)
if self.pattern.match(value) is None:
raise ValueError('Expected {}, not {}'.format(self.name, json_encode_string(value)))
return value
def format(self, value):
return None if value is None else six.text_type(value)
class OptionName(Validator):
""" Validates option names.
"""
pattern = re.compile(r'''(?=\w)[^\d]\w*$''', re.UNICODE)
def __call__(self, value):
if value is not None:
value = six.text_type(value)
if OptionName.pattern.match(value) is None:
raise ValueError('Illegal characters in option name: {}'.format(value))
return value
def format(self, value):
return None if value is None else six.text_type(value)
class RegularExpression(Validator):
""" Validates regular expression option values.
"""
def __call__(self, value):
if value is None:
return None
try:
value = re.compile(six.text_type(value))
except re.error as error:
raise ValueError('{}: {}'.format(six.text_type(error).capitalize(), value))
return value
def format(self, value):
return None if value is None else value.pattern
class Set(Validator):
""" Validates set option values.
"""
def __init__(self, *args):
self.membership = set(args)
def __call__(self, value):
if value is None:
return None
value = six.text_type(value)
if value not in self.membership:
raise ValueError('Unrecognized value: {}'.format(value))
return value
def format(self, value):
return self.__call__(value)
__all__ = ['Boolean', 'Code', 'Duration', 'File', 'Integer', 'List', 'Map', 'RegularExpression', 'Set']
dist/escu/bin/splunklib/six.py
Vendored
-980
View File
@@ -1,980 +0,0 @@
# Copyright (c) 2010-2020 Benjamin Peterson
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all
# copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
# SOFTWARE.
"""Utilities for writing code that runs on Python 2 and 3"""
from __future__ import absolute_import
import functools
import itertools
import operator
import sys
import types
__author__ = "Benjamin Peterson <benjamin@python.org>"
__version__ = "1.14.0"
# Useful for very coarse version differentiation.
PY2 = sys.version_info[0] == 2
PY3 = sys.version_info[0] == 3
PY34 = sys.version_info[0:2] >= (3, 4)
if PY3:
string_types = str,
integer_types = int,
class_types = type,
text_type = str
binary_type = bytes
MAXSIZE = sys.maxsize
else:
string_types = basestring,
integer_types = (int, long)
class_types = (type, types.ClassType)
text_type = unicode
binary_type = str
if sys.platform.startswith("java"):
# Jython always uses 32 bits.
MAXSIZE = int((1 << 31) - 1)
else:
# It's possible to have sizeof(long) != sizeof(Py_ssize_t).
class X(object):
def __len__(self):
return 1 << 31
try:
len(X())
except OverflowError:
# 32-bit
MAXSIZE = int((1 << 31) - 1)
else:
# 64-bit
MAXSIZE = int((1 << 63) - 1)
del X
def _add_doc(func, doc):
"""Add documentation to a function."""
func.__doc__ = doc
def _import_module(name):
"""Import module, returning the module after the last dot."""
__import__(name)
return sys.modules[name]
class _LazyDescr(object):
def __init__(self, name):
self.name = name
def __get__(self, obj, tp):
result = self._resolve()
setattr(obj, self.name, result) # Invokes __set__.
try:
# This is a bit ugly, but it avoids running this again by
# removing this descriptor.
delattr(obj.__class__, self.name)
except AttributeError:
pass
return result
class MovedModule(_LazyDescr):
def __init__(self, name, old, new=None):
super(MovedModule, self).__init__(name)
if PY3:
if new is None:
new = name
self.mod = new
else:
self.mod = old
def _resolve(self):
return _import_module(self.mod)
def __getattr__(self, attr):
_module = self._resolve()
value = getattr(_module, attr)
setattr(self, attr, value)
return value
class _LazyModule(types.ModuleType):
def __init__(self, name):
super(_LazyModule, self).__init__(name)
self.__doc__ = self.__class__.__doc__
def __dir__(self):
attrs = ["__doc__", "__name__"]
attrs += [attr.name for attr in self._moved_attributes]
return attrs
# Subclasses should override this
_moved_attributes = []
class MovedAttribute(_LazyDescr):
def __init__(self, name, old_mod, new_mod, old_attr=None, new_attr=None):
super(MovedAttribute, self).__init__(name)
if PY3:
if new_mod is None:
new_mod = name
self.mod = new_mod
if new_attr is None:
if old_attr is None:
new_attr = name
else:
new_attr = old_attr
self.attr = new_attr
else:
self.mod = old_mod
if old_attr is None:
old_attr = name
self.attr = old_attr
def _resolve(self):
module = _import_module(self.mod)
return getattr(module, self.attr)
class _SixMetaPathImporter(object):
"""
A meta path importer to import six.moves and its submodules.
This class implements a PEP302 finder and loader. It should be compatible
with Python 2.5 and all existing versions of Python3
"""
def __init__(self, six_module_name):
self.name = six_module_name
self.known_modules = {}
def _add_module(self, mod, *fullnames):
for fullname in fullnames:
self.known_modules[self.name + "." + fullname] = mod
def _get_module(self, fullname):
return self.known_modules[self.name + "." + fullname]
def find_module(self, fullname, path=None):
if fullname in self.known_modules:
return self
return None
def __get_module(self, fullname):
try:
return self.known_modules[fullname]
except KeyError:
raise ImportError("This loader does not know module " + fullname)
def load_module(self, fullname):
try:
# in case of a reload
return sys.modules[fullname]
except KeyError:
pass
mod = self.__get_module(fullname)
if isinstance(mod, MovedModule):
mod = mod._resolve()
else:
mod.__loader__ = self
sys.modules[fullname] = mod
return mod
def is_package(self, fullname):
"""
Return true, if the named module is a package.
We need this method to get correct spec objects with
Python 3.4 (see PEP451)
"""
return hasattr(self.__get_module(fullname), "__path__")
def get_code(self, fullname):
"""Return None
Required, if is_package is implemented"""
self.__get_module(fullname) # eventually raises ImportError
return None
get_source = get_code # same as get_code
_importer = _SixMetaPathImporter(__name__)
class _MovedItems(_LazyModule):
"""Lazy loading of moved objects"""
__path__ = [] # mark as package
_moved_attributes = [
MovedAttribute("cStringIO", "cStringIO", "io", "StringIO"),
MovedAttribute("filter", "itertools", "builtins", "ifilter", "filter"),
MovedAttribute("filterfalse", "itertools", "itertools", "ifilterfalse", "filterfalse"),
MovedAttribute("input", "__builtin__", "builtins", "raw_input", "input"),
MovedAttribute("intern", "__builtin__", "sys"),
MovedAttribute("map", "itertools", "builtins", "imap", "map"),
MovedAttribute("getcwd", "os", "os", "getcwdu", "getcwd"),
MovedAttribute("getcwdb", "os", "os", "getcwd", "getcwdb"),
MovedAttribute("getoutput", "commands", "subprocess"),
MovedAttribute("range", "__builtin__", "builtins", "xrange", "range"),
MovedAttribute("reload_module", "__builtin__", "importlib" if PY34 else "imp", "reload"),
MovedAttribute("reduce", "__builtin__", "functools"),
MovedAttribute("shlex_quote", "pipes", "shlex", "quote"),
MovedAttribute("StringIO", "StringIO", "io"),
MovedAttribute("UserDict", "UserDict", "collections"),
MovedAttribute("UserList", "UserList", "collections"),
MovedAttribute("UserString", "UserString", "collections"),
MovedAttribute("xrange", "__builtin__", "builtins", "xrange", "range"),
MovedAttribute("zip", "itertools", "builtins", "izip", "zip"),
MovedAttribute("zip_longest", "itertools", "itertools", "izip_longest", "zip_longest"),
MovedModule("builtins", "__builtin__"),
MovedModule("configparser", "ConfigParser"),
MovedModule("collections_abc", "collections", "collections.abc" if sys.version_info >= (3, 3) else "collections"),
MovedModule("copyreg", "copy_reg"),
MovedModule("dbm_gnu", "gdbm", "dbm.gnu"),
MovedModule("dbm_ndbm", "dbm", "dbm.ndbm"),
MovedModule("_dummy_thread", "dummy_thread", "_dummy_thread" if sys.version_info < (3, 9) else "_thread"),
MovedModule("http_cookiejar", "cookielib", "http.cookiejar"),
MovedModule("http_cookies", "Cookie", "http.cookies"),
MovedModule("html_entities", "htmlentitydefs", "html.entities"),
MovedModule("html_parser", "HTMLParser", "html.parser"),
MovedModule("http_client", "httplib", "http.client"),
MovedModule("email_mime_base", "email.MIMEBase", "email.mime.base"),
MovedModule("email_mime_image", "email.MIMEImage", "email.mime.image"),
MovedModule("email_mime_multipart", "email.MIMEMultipart", "email.mime.multipart"),
MovedModule("email_mime_nonmultipart", "email.MIMENonMultipart", "email.mime.nonmultipart"),
MovedModule("email_mime_text", "email.MIMEText", "email.mime.text"),
MovedModule("BaseHTTPServer", "BaseHTTPServer", "http.server"),
MovedModule("CGIHTTPServer", "CGIHTTPServer", "http.server"),
MovedModule("SimpleHTTPServer", "SimpleHTTPServer", "http.server"),
MovedModule("cPickle", "cPickle", "pickle"),
MovedModule("queue", "Queue"),
MovedModule("reprlib", "repr"),
MovedModule("socketserver", "SocketServer"),
MovedModule("_thread", "thread", "_thread"),
MovedModule("tkinter", "Tkinter"),
MovedModule("tkinter_dialog", "Dialog", "tkinter.dialog"),
MovedModule("tkinter_filedialog", "FileDialog", "tkinter.filedialog"),
MovedModule("tkinter_scrolledtext", "ScrolledText", "tkinter.scrolledtext"),
MovedModule("tkinter_simpledialog", "SimpleDialog", "tkinter.simpledialog"),
MovedModule("tkinter_tix", "Tix", "tkinter.tix"),
MovedModule("tkinter_ttk", "ttk", "tkinter.ttk"),
MovedModule("tkinter_constants", "Tkconstants", "tkinter.constants"),
MovedModule("tkinter_dnd", "Tkdnd", "tkinter.dnd"),
MovedModule("tkinter_colorchooser", "tkColorChooser",
"tkinter.colorchooser"),
MovedModule("tkinter_commondialog", "tkCommonDialog",
"tkinter.commondialog"),
MovedModule("tkinter_tkfiledialog", "tkFileDialog", "tkinter.filedialog"),
MovedModule("tkinter_font", "tkFont", "tkinter.font"),
MovedModule("tkinter_messagebox", "tkMessageBox", "tkinter.messagebox"),
MovedModule("tkinter_tksimpledialog", "tkSimpleDialog",
"tkinter.simpledialog"),
MovedModule("urllib_parse", __name__ + ".moves.urllib_parse", "urllib.parse"),
MovedModule("urllib_error", __name__ + ".moves.urllib_error", "urllib.error"),
MovedModule("urllib", __name__ + ".moves.urllib", __name__ + ".moves.urllib"),
MovedModule("urllib_robotparser", "robotparser", "urllib.robotparser"),
MovedModule("xmlrpc_client", "xmlrpclib", "xmlrpc.client"),
MovedModule("xmlrpc_server", "SimpleXMLRPCServer", "xmlrpc.server"),
]
# Add windows specific modules.
if sys.platform == "win32":
_moved_attributes += [
MovedModule("winreg", "_winreg"),
]
for attr in _moved_attributes:
setattr(_MovedItems, attr.name, attr)
if isinstance(attr, MovedModule):
_importer._add_module(attr, "moves." + attr.name)
del attr
_MovedItems._moved_attributes = _moved_attributes
moves = _MovedItems(__name__ + ".moves")
_importer._add_module(moves, "moves")
class Module_six_moves_urllib_parse(_LazyModule):
"""Lazy loading of moved objects in six.moves.urllib_parse"""
_urllib_parse_moved_attributes = [
MovedAttribute("ParseResult", "urlparse", "urllib.parse"),
MovedAttribute("SplitResult", "urlparse", "urllib.parse"),
MovedAttribute("parse_qs", "urlparse", "urllib.parse"),
MovedAttribute("parse_qsl", "urlparse", "urllib.parse"),
MovedAttribute("urldefrag", "urlparse", "urllib.parse"),
MovedAttribute("urljoin", "urlparse", "urllib.parse"),
MovedAttribute("urlparse", "urlparse", "urllib.parse"),
MovedAttribute("urlsplit", "urlparse", "urllib.parse"),
MovedAttribute("urlunparse", "urlparse", "urllib.parse"),
MovedAttribute("urlunsplit", "urlparse", "urllib.parse"),
MovedAttribute("quote", "urllib", "urllib.parse"),
MovedAttribute("quote_plus", "urllib", "urllib.parse"),
MovedAttribute("unquote", "urllib", "urllib.parse"),
MovedAttribute("unquote_plus", "urllib", "urllib.parse"),
MovedAttribute("unquote_to_bytes", "urllib", "urllib.parse", "unquote", "unquote_to_bytes"),
MovedAttribute("urlencode", "urllib", "urllib.parse"),
MovedAttribute("splitquery", "urllib", "urllib.parse"),
MovedAttribute("splittag", "urllib", "urllib.parse"),
MovedAttribute("splituser", "urllib", "urllib.parse"),
MovedAttribute("splitvalue", "urllib", "urllib.parse"),
MovedAttribute("uses_fragment", "urlparse", "urllib.parse"),
MovedAttribute("uses_netloc", "urlparse", "urllib.parse"),
MovedAttribute("uses_params", "urlparse", "urllib.parse"),
MovedAttribute("uses_query", "urlparse", "urllib.parse"),
MovedAttribute("uses_relative", "urlparse", "urllib.parse"),
]
for attr in _urllib_parse_moved_attributes:
setattr(Module_six_moves_urllib_parse, attr.name, attr)
del attr
Module_six_moves_urllib_parse._moved_attributes = _urllib_parse_moved_attributes
_importer._add_module(Module_six_moves_urllib_parse(__name__ + ".moves.urllib_parse"),
"moves.urllib_parse", "moves.urllib.parse")
class Module_six_moves_urllib_error(_LazyModule):
"""Lazy loading of moved objects in six.moves.urllib_error"""
_urllib_error_moved_attributes = [
MovedAttribute("URLError", "urllib2", "urllib.error"),
MovedAttribute("HTTPError", "urllib2", "urllib.error"),
MovedAttribute("ContentTooShortError", "urllib", "urllib.error"),
]
for attr in _urllib_error_moved_attributes:
setattr(Module_six_moves_urllib_error, attr.name, attr)
del attr
Module_six_moves_urllib_error._moved_attributes = _urllib_error_moved_attributes
_importer._add_module(Module_six_moves_urllib_error(__name__ + ".moves.urllib.error"),
"moves.urllib_error", "moves.urllib.error")
class Module_six_moves_urllib_request(_LazyModule):
"""Lazy loading of moved objects in six.moves.urllib_request"""
_urllib_request_moved_attributes = [
MovedAttribute("urlopen", "urllib2", "urllib.request"),
MovedAttribute("install_opener", "urllib2", "urllib.request"),
MovedAttribute("build_opener", "urllib2", "urllib.request"),
MovedAttribute("pathname2url", "urllib", "urllib.request"),
MovedAttribute("url2pathname", "urllib", "urllib.request"),
MovedAttribute("getproxies", "urllib", "urllib.request"),
MovedAttribute("Request", "urllib2", "urllib.request"),
MovedAttribute("OpenerDirector", "urllib2", "urllib.request"),
MovedAttribute("HTTPDefaultErrorHandler", "urllib2", "urllib.request"),
MovedAttribute("HTTPRedirectHandler", "urllib2", "urllib.request"),
MovedAttribute("HTTPCookieProcessor", "urllib2", "urllib.request"),
MovedAttribute("ProxyHandler", "urllib2", "urllib.request"),
MovedAttribute("BaseHandler", "urllib2", "urllib.request"),
MovedAttribute("HTTPPasswordMgr", "urllib2", "urllib.request"),
MovedAttribute("HTTPPasswordMgrWithDefaultRealm", "urllib2", "urllib.request"),
MovedAttribute("AbstractBasicAuthHandler", "urllib2", "urllib.request"),
MovedAttribute("HTTPBasicAuthHandler", "urllib2", "urllib.request"),
MovedAttribute("ProxyBasicAuthHandler", "urllib2", "urllib.request"),
MovedAttribute("AbstractDigestAuthHandler", "urllib2", "urllib.request"),
MovedAttribute("HTTPDigestAuthHandler", "urllib2", "urllib.request"),
MovedAttribute("ProxyDigestAuthHandler", "urllib2", "urllib.request"),
MovedAttribute("HTTPHandler", "urllib2", "urllib.request"),
MovedAttribute("HTTPSHandler", "urllib2", "urllib.request"),
MovedAttribute("FileHandler", "urllib2", "urllib.request"),
MovedAttribute("FTPHandler", "urllib2", "urllib.request"),
MovedAttribute("CacheFTPHandler", "urllib2", "urllib.request"),
MovedAttribute("UnknownHandler", "urllib2", "urllib.request"),
MovedAttribute("HTTPErrorProcessor", "urllib2", "urllib.request"),
MovedAttribute("urlretrieve", "urllib", "urllib.request"),
MovedAttribute("urlcleanup", "urllib", "urllib.request"),
MovedAttribute("URLopener", "urllib", "urllib.request"),
MovedAttribute("FancyURLopener", "urllib", "urllib.request"),
MovedAttribute("proxy_bypass", "urllib", "urllib.request"),
MovedAttribute("parse_http_list", "urllib2", "urllib.request"),
MovedAttribute("parse_keqv_list", "urllib2", "urllib.request"),
]
for attr in _urllib_request_moved_attributes:
setattr(Module_six_moves_urllib_request, attr.name, attr)
del attr
Module_six_moves_urllib_request._moved_attributes = _urllib_request_moved_attributes
_importer._add_module(Module_six_moves_urllib_request(__name__ + ".moves.urllib.request"),
"moves.urllib_request", "moves.urllib.request")
class Module_six_moves_urllib_response(_LazyModule):
"""Lazy loading of moved objects in six.moves.urllib_response"""
_urllib_response_moved_attributes = [
MovedAttribute("addbase", "urllib", "urllib.response"),
MovedAttribute("addclosehook", "urllib", "urllib.response"),
MovedAttribute("addinfo", "urllib", "urllib.response"),
MovedAttribute("addinfourl", "urllib", "urllib.response"),
]
for attr in _urllib_response_moved_attributes:
setattr(Module_six_moves_urllib_response, attr.name, attr)
del attr
Module_six_moves_urllib_response._moved_attributes = _urllib_response_moved_attributes
_importer._add_module(Module_six_moves_urllib_response(__name__ + ".moves.urllib.response"),
"moves.urllib_response", "moves.urllib.response")
class Module_six_moves_urllib_robotparser(_LazyModule):
"""Lazy loading of moved objects in six.moves.urllib_robotparser"""
_urllib_robotparser_moved_attributes = [
MovedAttribute("RobotFileParser", "robotparser", "urllib.robotparser"),
]
for attr in _urllib_robotparser_moved_attributes:
setattr(Module_six_moves_urllib_robotparser, attr.name, attr)
del attr
Module_six_moves_urllib_robotparser._moved_attributes = _urllib_robotparser_moved_attributes
_importer._add_module(Module_six_moves_urllib_robotparser(__name__ + ".moves.urllib.robotparser"),
"moves.urllib_robotparser", "moves.urllib.robotparser")
class Module_six_moves_urllib(types.ModuleType):
"""Create a six.moves.urllib namespace that resembles the Python 3 namespace"""
__path__ = [] # mark as package
parse = _importer._get_module("moves.urllib_parse")
error = _importer._get_module("moves.urllib_error")
request = _importer._get_module("moves.urllib_request")
response = _importer._get_module("moves.urllib_response")
robotparser = _importer._get_module("moves.urllib_robotparser")
def __dir__(self):
return ['parse', 'error', 'request', 'response', 'robotparser']
_importer._add_module(Module_six_moves_urllib(__name__ + ".moves.urllib"),
"moves.urllib")
def add_move(move):
"""Add an item to six.moves."""
setattr(_MovedItems, move.name, move)
def remove_move(name):
"""Remove item from six.moves."""
try:
delattr(_MovedItems, name)
except AttributeError:
try:
del moves.__dict__[name]
except KeyError:
raise AttributeError("no such move, %r" % (name,))
if PY3:
_meth_func = "__func__"
_meth_self = "__self__"
_func_closure = "__closure__"
_func_code = "__code__"
_func_defaults = "__defaults__"
_func_globals = "__globals__"
else:
_meth_func = "im_func"
_meth_self = "im_self"
_func_closure = "func_closure"
_func_code = "func_code"
_func_defaults = "func_defaults"
_func_globals = "func_globals"
try:
advance_iterator = next
except NameError:
def advance_iterator(it):
return it.next()
next = advance_iterator
try:
callable = callable
except NameError:
def callable(obj):
return any("__call__" in klass.__dict__ for klass in type(obj).__mro__)
if PY3:
def get_unbound_function(unbound):
return unbound
create_bound_method = types.MethodType
def create_unbound_method(func, cls):
return func
Iterator = object
else:
def get_unbound_function(unbound):
return unbound.im_func
def create_bound_method(func, obj):
return types.MethodType(func, obj, obj.__class__)
def create_unbound_method(func, cls):
return types.MethodType(func, None, cls)
class Iterator(object):
def next(self):
return type(self).__next__(self)
callable = callable
_add_doc(get_unbound_function,
"""Get the function out of a possibly unbound function""")
get_method_function = operator.attrgetter(_meth_func)
get_method_self = operator.attrgetter(_meth_self)
get_function_closure = operator.attrgetter(_func_closure)
get_function_code = operator.attrgetter(_func_code)
get_function_defaults = operator.attrgetter(_func_defaults)
get_function_globals = operator.attrgetter(_func_globals)
if PY3:
def iterkeys(d, **kw):
return iter(d.keys(**kw))
def itervalues(d, **kw):
return iter(d.values(**kw))
def iteritems(d, **kw):
return iter(d.items(**kw))
def iterlists(d, **kw):
return iter(d.lists(**kw))
viewkeys = operator.methodcaller("keys")
viewvalues = operator.methodcaller("values")
viewitems = operator.methodcaller("items")
else:
def iterkeys(d, **kw):
return d.iterkeys(**kw)
def itervalues(d, **kw):
return d.itervalues(**kw)
def iteritems(d, **kw):
return d.iteritems(**kw)
def iterlists(d, **kw):
return d.iterlists(**kw)
viewkeys = operator.methodcaller("viewkeys")
viewvalues = operator.methodcaller("viewvalues")
viewitems = operator.methodcaller("viewitems")
_add_doc(iterkeys, "Return an iterator over the keys of a dictionary.")
_add_doc(itervalues, "Return an iterator over the values of a dictionary.")
_add_doc(iteritems,
"Return an iterator over the (key, value) pairs of a dictionary.")
_add_doc(iterlists,
"Return an iterator over the (key, [values]) pairs of a dictionary.")
if PY3:
def b(s):
return s.encode("latin-1")
def u(s):
return s
unichr = chr
import struct
int2byte = struct.Struct(">B").pack
del struct
byte2int = operator.itemgetter(0)
indexbytes = operator.getitem
iterbytes = iter
import io
StringIO = io.StringIO
BytesIO = io.BytesIO
del io
_assertCountEqual = "assertCountEqual"
if sys.version_info[1] <= 1:
_assertRaisesRegex = "assertRaisesRegexp"
_assertRegex = "assertRegexpMatches"
_assertNotRegex = "assertNotRegexpMatches"
else:
_assertRaisesRegex = "assertRaisesRegex"
_assertRegex = "assertRegex"
_assertNotRegex = "assertNotRegex"
else:
def b(s):
return s
# Workaround for standalone backslash
def u(s):
return unicode(s.replace(r'\\', r'\\\\'), "unicode_escape")
unichr = unichr
int2byte = chr
def byte2int(bs):
return ord(bs[0])
def indexbytes(buf, i):
return ord(buf[i])
iterbytes = functools.partial(itertools.imap, ord)
import StringIO
StringIO = BytesIO = StringIO.StringIO
_assertCountEqual = "assertItemsEqual"
_assertRaisesRegex = "assertRaisesRegexp"
_assertRegex = "assertRegexpMatches"
_assertNotRegex = "assertNotRegexpMatches"
_add_doc(b, """Byte literal""")
_add_doc(u, """Text literal""")
def assertCountEqual(self, *args, **kwargs):
return getattr(self, _assertCountEqual)(*args, **kwargs)
def assertRaisesRegex(self, *args, **kwargs):
return getattr(self, _assertRaisesRegex)(*args, **kwargs)
def assertRegex(self, *args, **kwargs):
return getattr(self, _assertRegex)(*args, **kwargs)
def assertNotRegex(self, *args, **kwargs):
return getattr(self, _assertNotRegex)(*args, **kwargs)
if PY3:
exec_ = getattr(moves.builtins, "exec")
def reraise(tp, value, tb=None):
try:
if value is None:
value = tp()
if value.__traceback__ is not tb:
raise value.with_traceback(tb)
raise value
finally:
value = None
tb = None
else:
def exec_(_code_, _globs_=None, _locs_=None):
"""Execute code in a namespace."""
if _globs_ is None:
frame = sys._getframe(1)
_globs_ = frame.f_globals
if _locs_ is None:
_locs_ = frame.f_locals
del frame
elif _locs_ is None:
_locs_ = _globs_
exec("""exec _code_ in _globs_, _locs_""")
exec_("""def reraise(tp, value, tb=None):
try:
raise tp, value, tb
finally:
tb = None
""")
if sys.version_info[:2] > (3,):
exec_("""def raise_from(value, from_value):
try:
raise value from from_value
finally:
value = None
""")
else:
def raise_from(value, from_value):
raise value
print_ = getattr(moves.builtins, "print", None)
if print_ is None:
def print_(*args, **kwargs):
"""The new-style print function for Python 2.4 and 2.5."""
fp = kwargs.pop("file", sys.stdout)
if fp is None:
return
def write(data):
if not isinstance(data, basestring):
data = str(data)
# If the file has an encoding, encode unicode with it.
if (isinstance(fp, file) and
isinstance(data, unicode) and
fp.encoding is not None):
errors = getattr(fp, "errors", None)
if errors is None:
errors = "strict"
data = data.encode(fp.encoding, errors)
fp.write(data)
want_unicode = False
sep = kwargs.pop("sep", None)
if sep is not None:
if isinstance(sep, unicode):
want_unicode = True
elif not isinstance(sep, str):
raise TypeError("sep must be None or a string")
end = kwargs.pop("end", None)
if end is not None:
if isinstance(end, unicode):
want_unicode = True
elif not isinstance(end, str):
raise TypeError("end must be None or a string")
if kwargs:
raise TypeError("invalid keyword arguments to print()")
if not want_unicode:
for arg in args:
if isinstance(arg, unicode):
want_unicode = True
break
if want_unicode:
newline = unicode("\n")
space = unicode(" ")
else:
newline = "\n"
space = " "
if sep is None:
sep = space
if end is None:
end = newline
for i, arg in enumerate(args):
if i:
write(sep)
write(arg)
write(end)
if sys.version_info[:2] < (3, 3):
_print = print_
def print_(*args, **kwargs):
fp = kwargs.get("file", sys.stdout)
flush = kwargs.pop("flush", False)
_print(*args, **kwargs)
if flush and fp is not None:
fp.flush()
_add_doc(reraise, """Reraise an exception.""")
if sys.version_info[0:2] < (3, 4):
# This does exactly the same what the :func:`py3:functools.update_wrapper`
# function does on Python versions after 3.2. It sets the ``__wrapped__``
# attribute on ``wrapper`` object and it doesn't raise an error if any of
# the attributes mentioned in ``assigned`` and ``updated`` are missing on
# ``wrapped`` object.
def _update_wrapper(wrapper, wrapped,
assigned=functools.WRAPPER_ASSIGNMENTS,
updated=functools.WRAPPER_UPDATES):
for attr in assigned:
try:
value = getattr(wrapped, attr)
except AttributeError:
continue
else:
setattr(wrapper, attr, value)
for attr in updated:
getattr(wrapper, attr).update(getattr(wrapped, attr, {}))
wrapper.__wrapped__ = wrapped
return wrapper
_update_wrapper.__doc__ = functools.update_wrapper.__doc__
def wraps(wrapped, assigned=functools.WRAPPER_ASSIGNMENTS,
updated=functools.WRAPPER_UPDATES):
return functools.partial(_update_wrapper, wrapped=wrapped,
assigned=assigned, updated=updated)
wraps.__doc__ = functools.wraps.__doc__
else:
wraps = functools.wraps
def with_metaclass(meta, *bases):
"""Create a base class with a metaclass."""
# This requires a bit of explanation: the basic idea is to make a dummy
# metaclass for one level of class instantiation that replaces itself with
# the actual metaclass.
class metaclass(type):
def __new__(cls, name, this_bases, d):
if sys.version_info[:2] >= (3, 7):
# This version introduced PEP 560 that requires a bit
# of extra care (we mimic what is done by __build_class__).
resolved_bases = types.resolve_bases(bases)
if resolved_bases is not bases:
d['__orig_bases__'] = bases
else:
resolved_bases = bases
return meta(name, resolved_bases, d)
@classmethod
def __prepare__(cls, name, this_bases):
return meta.__prepare__(name, bases)
return type.__new__(metaclass, 'temporary_class', (), {})
def add_metaclass(metaclass):
"""Class decorator for creating a class with a metaclass."""
def wrapper(cls):
orig_vars = cls.__dict__.copy()
slots = orig_vars.get('__slots__')
if slots is not None:
if isinstance(slots, str):
slots = [slots]
for slots_var in slots:
orig_vars.pop(slots_var)
orig_vars.pop('__dict__', None)
orig_vars.pop('__weakref__', None)
if hasattr(cls, '__qualname__'):
orig_vars['__qualname__'] = cls.__qualname__
return metaclass(cls.__name__, cls.__bases__, orig_vars)
return wrapper
def ensure_binary(s, encoding='utf-8', errors='strict'):
"""Coerce **s** to six.binary_type.
For Python 2:
- `unicode` -> encoded to `str`
- `str` -> `str`
For Python 3:
- `str` -> encoded to `bytes`
- `bytes` -> `bytes`
"""
if isinstance(s, text_type):
return s.encode(encoding, errors)
elif isinstance(s, binary_type):
return s
else:
raise TypeError("not expecting type '%s'" % type(s))
def ensure_str(s, encoding='utf-8', errors='strict'):
"""Coerce *s* to `str`.
For Python 2:
- `unicode` -> encoded to `str`
- `str` -> `str`
For Python 3:
- `str` -> `str`
- `bytes` -> decoded to `str`
"""
if not isinstance(s, (text_type, binary_type)):
raise TypeError("not expecting type '%s'" % type(s))
if PY2 and isinstance(s, text_type):
s = s.encode(encoding, errors)
elif PY3 and isinstance(s, binary_type):
s = s.decode(encoding, errors)
return s
def ensure_text(s, encoding='utf-8', errors='strict'):
"""Coerce *s* to six.text_type.
For Python 2:
- `unicode` -> `unicode`
- `str` -> `unicode`
For Python 3:
- `str` -> `str`
- `bytes` -> decoded to `str`
"""
if isinstance(s, binary_type):
return s.decode(encoding, errors)
elif isinstance(s, text_type):
return s
else:
raise TypeError("not expecting type '%s'" % type(s))
def python_2_unicode_compatible(klass):
"""
A class decorator that defines __unicode__ and __str__ methods under Python 2.
Under Python 3 it does nothing.
To support Python 2 and 3 with a single code base, define a __str__ method
returning text and apply this decorator to the class.
"""
if PY2:
if '__str__' not in klass.__dict__:
raise ValueError("@python_2_unicode_compatible cannot be applied "
"to %s because it doesn't define __str__()." %
klass.__name__)
klass.__unicode__ = klass.__str__
klass.__str__ = lambda self: self.__unicode__().encode('utf-8')
return klass
# Complete the moves implementation.
# This code is at the end of this module to speed up module loading.
# Turn this module into a package.
__path__ = [] # required for PEP 302 and PEP 451
__package__ = __name__ # see PEP 366 @ReservedAssignment
if globals().get("__spec__") is not None:
__spec__.submodule_search_locations = [] # PEP 451 @UndefinedVariable
# Remove other six meta path importers, since they cause problems. This can
# happen if six is removed from sys.modules and then reloaded. (Setuptools does
# this for some reason.)
if sys.meta_path:
for i, importer in enumerate(sys.meta_path):
# Here's some real nastiness: Another "instance" of the six module might
# be floating around. Therefore, we can't use isinstance() to check for
# the six meta path importer, since the other six instance will have
# inserted an importer with different class.
if (type(importer).__name__ == "_SixMetaPathImporter" and
importer.name == __name__):
del sys.meta_path[i]
break
del i, importer
# Finally, add the importer to the meta path import hook.
sys.meta_path.append(_importer)
stories/f5_big_ip_vulnerability_cve_2022_1388.yml
+24
View File
@@ -0,0 +1,24 @@
name: F5 BIG-IP Vulnerability CVE-2022-1388
id: 0367b177-f8d6-4c4b-a62d-86f52a590bff
version: 1
date: '2022-05-10'
author: Michael Haag, Splunk
description: CVE-2022-1388 is a unauthenticated remote code execution vulnerablity against BIG-IP iControl REST API.
narrative: CVE-2022-1388 is a critical vulnerability (CVSS 9.8) in the management interface of F5 Networks'' BIG-IP solution that enables an unauthenticated attacker to gain remote code execution on the system through bypassing F5''s iControl REST authentication. The vulnerability was first discovered by F5''s internal product security team and disclosed publicly on May 4, 2022, per Randori.
This vulnerability,CVE-2022-1388, may allow an unauthenticated attacker with network access to the BIG-IP system through the management port and/or self IP addresses to execute arbitrary system commands, create or delete files, or disable services. There is no data plane exposure; this is a control plane issue only per F5 article K23605346.
Is CVE-2022-1388 Exploitable? Yes. There are now multiple POC scripts available and reports of threat actors scanning and potentially exploiting the vulnerablity. Per Randori the specific interface needed to exploit this vulnerability is rarely publicly exposed, and the risk to most organizations of exploitation by an unauthenticated external actor is low.
references:
- https://github.com/dk4trin/templates-nuclei/blob/main/CVE-2022-1388.yaml
- https://www.randori.com/blog/vulnerability-analysis-cve-2022-1388/
- https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-1388
- https://twitter.com/da_667/status/1523770267327250438?s=20&t=-JnB_aNWuJFsmcOmxGUWLQ
- https://github.com/horizon3ai/CVE-2022-1388/blob/main/CVE-2022-1388.py
tags:
analytic_story: F5 BIG-IP Vulnerability CVE-2022-1388
category:
- Adversary Tactics
product:
- Splunk Enterprise
- Splunk Enterprise Security
- Splunk Cloud
usecase: Advanced Threat Detection
tests/endpoint/get_addefaultdomainpasswordpolicy_with_powershell_script_block.test.yml
+4 -4
View File
@@ -6,7 +6,7 @@ tests:
earliest_time: -24h
latest_time: now
attack_data:
- file_name: windows-powershell.log
data: https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1201/pwd_policy_discovery/windows-powershell.log
source: WinEventLog:Microsoft-Windows-PowerShell/Operational
sourcetype: WinEventLog
- file_name: windows-powershell-xml.log
data: https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1201/pwd_policy_discovery/windows-powershell-xml.log
source: XmlWinEventLog:Microsoft-Windows-PowerShell/Operational
sourcetype: XmlWinEventLog
tests/endpoint/get_domainuser_with_powershell_script_block.test.yml
+4 -4
View File
@@ -6,7 +6,7 @@ tests:
earliest_time: -24h
latest_time: now
attack_data:
- file_name: windows-powershell.log
data: https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1087.002/AD_discovery/windows-powershell.log
source: WinEventLog:Microsoft-Windows-PowerShell/Operational
sourcetype: WinEventLog
- file_name: windows-powershell-xml.log
data: https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1087.002/AD_discovery/windows-powershell-xml.log
source: XmlWinEventLog:Microsoft-Windows-PowerShell/Operational
sourcetype: XmlWinEventLog
tests/endpoint/get_foresttrust_with_powershell_script_block.test.yml
+4 -4
View File
@@ -6,7 +6,7 @@ tests:
earliest_time: -24h
latest_time: now
attack_data:
- file_name: windows-powershell.log
data: https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1482/discovery/windows-powershell.log
source: WinEventLog:Microsoft-Windows-PowerShell/Operational
sourcetype: WinEventLog:Microsoft-Windows-PowerShell/Operational
- file_name: windows-powershell-xml.log
data: https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1482/discovery/windows-powershell-xml.log
source: XmlWinEventLog:Microsoft-Windows-PowerShell/Operational
sourcetype: XmlWinEventLog
tests/endpoint/get_wmiobject_group_discovery_with_script_block_logging.test.yml
+4 -4
View File
@@ -6,7 +6,7 @@ tests:
earliest_time: -24h
latest_time: now
attack_data:
- file_name: windows-powershell.log
data: https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1069.001/atomic_red_team/windows-powershell.log
source: WinEventLog:Microsoft-Windows-PowerShell/Operational
sourcetype: wineventlog
- file_name: windows-powershell-xml.log
data: https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1069.001/atomic_red_team/windows-powershell-xml.log
source: XmlWinEventLog:Microsoft-Windows-PowerShell/Operational
sourcetype: xmlwineventlog
tests/endpoint/getadgroup_with_powershell_script_block.test.yml
+4 -4
View File
@@ -6,7 +6,7 @@ tests:
earliest_time: -24h
latest_time: now
attack_data:
- file_name: windows-powershell.log
data: https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1069.002/AD_discovery/windows-powershell.log
source: WinEventLog:Microsoft-Windows-PowerShell/Operational
sourcetype: wineventlog
- file_name: windows-powershell-xml.log
data: https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1069.002/AD_discovery/windows-powershell-xml.log
source: XmlWinEventLog:Microsoft-Windows-PowerShell/Operational
sourcetype: xmlwineventlog
tests/endpoint/getcurrent_user_with_powershell_script_block.test.yml
+4 -4
View File
@@ -6,7 +6,7 @@ tests:
earliest_time: -24h
latest_time: now
attack_data:
- file_name: windows-powershell.log
data: https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1033/AD_discovery/windows-powershell.log
source: WinEventLog:Microsoft-Windows-PowerShell/Operational
sourcetype: wineventlog
- file_name: windows-powershell-xml.log
data: https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1033/AD_discovery/windows-powershell-xml.log
source: XmlWinEventLog:Microsoft-Windows-PowerShell/Operational
sourcetype: xmlwineventlog
tests/endpoint/getlocaluser_with_powershell_script_block.test.yml
+4 -4
View File
@@ -6,7 +6,7 @@ tests:
earliest_time: -24h
latest_time: now
attack_data:
- file_name: windows-powershell.log
data: https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1087.001/AD_discovery/windows-powershell.log
source: WinEventLog:Microsoft-Windows-PowerShell/Operational
sourcetype: wineventlog
- file_name: windows-powershell-xml.log
data: https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1087.001/AD_discovery/windows-powershell-xml.log
source: XmlWinEventLog:Microsoft-Windows-PowerShell/Operational
sourcetype: XmlWinEventLog
tests/endpoint/interactive_session_on_remote_endpoint_with_powershell.test.yml
+4 -4
View File
@@ -6,7 +6,7 @@ tests:
earliest_time: -24h
latest_time: now
attack_data:
- file_name: windows-powershell.log
data: https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1021.006/lateral_movement_pssession/windows-powershell.log
source: WinEventLog:Microsoft-Windows-PowerShell/Operational
sourcetype: wineventlog
- file_name: windows-powershell-xml.log
data: https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1021.006/lateral_movement_pssession/windows-powershell-xml.log
source: XmlWinEventLog:Microsoft-Windows-PowerShell/Operational
sourcetype: xmlwineventlog
tests/endpoint/kerberos_pre_authentication_flag_disabled_with_powershell.test.yml
+4 -4
View File
@@ -6,8 +6,8 @@ tests:
earliest_time: '-24h'
latest_time: 'now'
attack_data:
- file_name: windows-powershell.log
data: https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1558.004/powershell/windows-powershell.log
source: WinEventLog:Microsoft-Windows-PowerShell/Operational
sourcetype: wineventlog
- file_name: windows-powershell-xml.log
data: https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1558.004/powershell/windows-powershell-xml.log
source: XmlWinEventLog:Microsoft-Windows-PowerShell/Operational
sourcetype: xmlwineventlog
update_timestamp: True
tests/endpoint/linux_deletion_of_ssh_key.test.yml → tests/endpoint/linux_deletion_of_ssh_hash_conf.test.yml
+3 -3
View File
@@ -1,7 +1,7 @@
name: Linux deletion Of SSH Key Unit Test
name: Linux deletion Of SSH Hash Conf Unit Test
tests:
- name: Linux deletion Of SSH Key
file: endpoint/linux_deletion_of_ssh_key.yml
- name: Linux deletion Of SSH Hash Conf
file: endpoint/linux_deletion_of_ssh_hash_conf.yml
pass_condition: '| stats count | where count > 0'
earliest_time: -24h
latest_time: now
tests/endpoint/powershell_execute_com_object.test.yml
+4 -4
View File
@@ -6,7 +6,7 @@ tests:
earliest_time: -24h
latest_time: now
attack_data:
- file_name: windows-powershell.log
data: https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/malware/conti/conti_leak/windows-powershell.log
source: WinEventLog:Microsoft-Windows-PowerShell/Operational
sourcetype: WinEventLog
- file_name: windows-powershell-xml.log
data: https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1546.015/pwh_com_object/windows-powershell-xml.log
source: XmlWinEventLog:Microsoft-Windows-PowerShell/Operational
sourcetype: xmlwineventlog
tests/endpoint/powershell_using_memory_as_backing_store.test.yml
+4 -4
View File
@@ -6,7 +6,7 @@ tests:
earliest_time: -24h
latest_time: now
attack_data:
- file_name: windows-powershell.log
data: https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/honeypots/pwsh/windows-powershell.log
source: WinEventLog:Microsoft-Windows-PowerShell/Operational
sourcetype: WinEventLog
- file_name: windows-powershell-xml.log
data: https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/honeypots/pwsh/windows-powershell-xml.log
source: XmlWinEventLog:Microsoft-Windows-PowerShell/Operational
sourcetype: xmlwineventlog
tests/endpoint/recon_avproduct_through_pwh_or_wmi.test.yml
+4 -4
View File
@@ -6,7 +6,7 @@ tests:
earliest_time: -24h
latest_time: now
attack_data:
- file_name: windows-powershell.log
data: https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/honeypots/pwsh/windows-powershell.log
source: WinEventLog:Microsoft-Windows-PowerShell/Operational
sourcetype: WinEventLog
- file_name: windows-powershell-xml.log
data: https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/t1592/pwh_av_recon/windows-powershell-xml.log
source: XmlWinEventLog:Microsoft-Windows-PowerShell/Operational
sourcetype: xmlwineventlog
tests/endpoint/remote_process_instantiation_via_dcom_and_powershell_script_block.test.yml
+4 -4
View File
@@ -6,7 +6,7 @@ tests:
earliest_time: -24h
latest_time: now
attack_data:
- file_name: windows-powershell.log
data: https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1021.003/lateral_movement/windows-powershell.log
source: WinEventLog:Microsoft-Windows-PowerShell/Operational
sourcetype: wineventlog
- file_name: windows-powershell-xml.log
data: https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1021.006/lateral_movement_psh/windows-powershell-xml.log
source: XmlWinEventLog:Microsoft-Windows-PowerShell/Operational
sourcetype: xmlwineventlog
tests/endpoint/remote_process_instantiation_via_winrm_and_powershell_script_block.test.yml
+4 -4
View File
@@ -6,7 +6,7 @@ tests:
earliest_time: -24h
latest_time: now
attack_data:
- file_name: windows-powershell.log
data: https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1021.006/lateral_movement_psh/windows-powershell.log
source: WinEventLog:Microsoft-Windows-PowerShell/Operational
sourcetype: wineventlog
- file_name: windows-powershell-xml.log
data: https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1021.006/lateral_movement_psh/windows-powershell-xml.log
source: XmlWinEventLog:Microsoft-Windows-PowerShell/Operational
sourcetype: xmlwineventlog
tests/endpoint/unloading_amsi_via_reflection.test.yml
+4 -4
View File
@@ -6,7 +6,7 @@ tests:
earliest_time: -24h
latest_time: now
attack_data:
- file_name: windows-powershell.log
data: https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1059.001/powershell_script_block_logging/windows-powershell.log
source: WinEventLog:Microsoft-Windows-PowerShell/Operational
sourcetype: WinEventLog
- file_name: windows-powershell-xml.log
data: https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1059.001/powershell_script_block_logging/windows-powershell-xml.log
source: XmlWinEventLog:Microsoft-Windows-PowerShell/Operational
sourcetype: xmlwineventlog
tests/endpoint/user_discovery_with_env_vars_powershell_script_block.test.yml
+4 -4
View File
@@ -6,7 +6,7 @@ tests:
earliest_time: -24h
latest_time: now
attack_data:
- file_name: windows-powershell.log
data: https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1033/AD_discovery/windows-powershell.log
source: WinEventLog:Microsoft-Windows-PowerShell/Operational
sourcetype: wineventlog
- file_name: windows-powershell-xml.log
data: https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1033/AD_discovery/windows-powershell-xml.log
source: XmlWinEventLog:Microsoft-Windows-PowerShell/Operational
sourcetype: xmlwineventlog
tests/network/f5_big_ip_icontrol_rest_vulnerability_cve_2022_1388.test.yml
+13
View File
@@ -0,0 +1,13 @@
name: F5 BIG-IP iControl REST Vulnerability CVE-2022-1388 Unit Test
tests:
- name: F5 BIG-IP iControl REST Vulnerability CVE-2022-1388
file: network/f5_big_ip_icontrol_rest_vulnerability_cve_2022_1388.yml
pass_condition: '| stats count | where count > 0'
earliest_time: -24h
latest_time: now
attack_data:
- file_name: f5.log
data: https://media.githubusercontent.com/media/splunk/attack_data/master/datasets/attack_techniques/T1190/f5/f5.log
source: pan:threat
sourcetype: pan:threat
update_timestamp: true