diff --git a/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml b/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml index 65d30d977a..4237449026 100644 --- a/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml +++ b/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml @@ -1,25 +1,33 @@ name: Detect DGA domains using pretrained model in DSDL -id: 92e24f32-9b9a-4060-bba2-2a0eb3bf3493 +id: 92e24f32-9b9a-4060-bba2-2a0eb31f3493 version: 1 -date: '2022-09-15' +date: "2023-01-18" author: Abhinav Mishra, Kumar Sharad and Namratha Sreekanta, Splunk type: Anomaly datamodel: - Network_Resolution -description: 'The following analytic uses a pretrained deep learning model to detect Domain Generation Algorithm (DGA) generated domains. The model is trained independently and is then made available for download. One of the prominent indicators of a domain being DGA generated is if the domain name consists of unusual character sequences or concatenated dictionary words. Adversaries often use clever techniques to obfuscate machine generated domain names as human generated. Predicting DGA generated domain names requires analysis and building a model based on carefully chosen features. The deep learning model we have developed uses the domain name to anlayze patterns of character sequences along with carefully chosen custom features to predict if a domain is DGA generated. The model takes a domain name consisting of second-level and top-level domain names as input and outputs a dga_score. Higher the dga_score, the more likely the input domain is a DGA domain. The threshold for flagging a domain as DGA is set at 0.5.' -search: '| tstats `security_content_summariesonly` values(DNS.answer) as IPs min(_time) as firstTime max(_time) as lastTime from datamodel=Network_Resolution by DNS.src, DNS.query -| `drop_dm_object_name("DNS")` | rename query AS domain | fields IPs, src, domain, firstTime, lastTime | apply pretrained_dga_model_dsdl | rename pred_dga_proba AS dga_score | where dga_score>0.5 | `security_content_ctime(firstTime)` | `security_content_ctime(lastTime)` | table src, domain, IPs, firstTime, lastTime, dga_score | `detect_dga_domains_using_pretrained_model_in_dsdl_filter`' -how_to_implement: " -* Steps to deploy DGA detection model into Splunk App DSDL. This detection depends on the Splunk app for Data Science and Deep Learning which can be found here - https://splunkbase.splunk.com/app/4607/ and the Network Resolution datamodel which can be found here - https://splunkbase.splunk.com/app/1621/. The detection uses a pre-trained deep learning model that needs to be deployed in DSDL app. Follow the steps for deployment here - https://github.com/splunk/security_content/wiki/How-to-deploy-pre-trained-Deep-Learning-models-for-ESCU. -* Download the artifacts .tar.gz file from the link - https://seal.splunkresearch.com/pretrained_dga_model_dsdl.tar.gz -* Download the pretrained_dga_model_dsdl.ipynb Jupyter notebook from https://github.com/splunk/security_content/notebooks -* Login to the Jupyter Lab for pretrained_dga_model_dsdl container. This container should be listed on Containers page for DSDL app. -* Below steps need to be followed inside Jupyter lab - * Upload the pretrained_dga_model_dsdl.tar.gz file into app/model/data path using the upload option in the jupyter notebook. - * Untar the artifact pretrained_dga_model_dsdl.tar.gz using tar -xf app/model/data/pretrained_dga_model_dsdl.tar.gz -C app/model/data - * Upload pretrained_dga_model_dsdl.pynb into Jupyter lab notebooks folder using the upload option in Jupyter lab - * Save the notebook using the save option in jupyter notebook. - * Upload pretrained_dga_model_dsdl.json into notebooks/data folder." +description: The following analytic uses a pre trained deep learning model to detect Domain Generation Algorithm (DGA) generated domains. The model is trained independently and is then made available for download. One of the prominent indicators of a domain being DGA generated is if the domain name consists of unusual character sequences or concatenated dictionary words. Adversaries often use clever techniques to obfuscate machine generated domain names as human generated. Predicting DGA generated domain names requires analysis and building a model based on carefully chosen features. The deep learning model we have developed uses the domain name to analyze patterns of character sequences along with carefully chosen custom features to predict if a domain is DGA generated. The model takes a domain name consisting of second-level and top-level domain names as input and outputs a dga_score. Higher the dga_score, the more likely the input domain is a DGA domain. The threshold for flagging a domain as DGA is set at 0.5. +search: "| tstats `security_content_summariesonly` values(DNS.answer) as IPs min(_time) as firstTime max(_time) as lastTime from datamodel=Network_Resolution by DNS.src, DNS.query | `drop_dm_object_name(DNS)` | rename query AS domain | fields IPs, src, domain, firstTime, lastTime | apply pretrained_dga_model_dsdl | rename pred_dga_proba AS dga_score | where dga_score>0.5 | `security_content_ctime(firstTime)` | `security_content_ctime(lastTime)` | table src, domain, IPs, firstTime, lastTime, dga_score | `detect_dga_domains_using_pretrained_model_in_dsdl_filter`" + +how_to_implement: 'Steps to deploy DGA detection model into Splunk App DSDL.\ + This detection depends on the Splunk app for Data Science and Deep Learning which can be found here - https://splunkbase.splunk.com/app/4607/ and the Network Resolution datamodel which can be found here - https://splunkbase.splunk.com/app/1621/. The detection uses a pre-trained deep learning model that needs to be deployed in DSDL app. Follow the steps for deployment here - https://github.com/splunk/security_content/wiki/How-to-deploy-pre-trained-Deep-Learning-models-for-ESCU.\ +* Download the artifacts .tar.gz file from the link `https://seal.splunkresearch.com/pretrained_dga_model_dsdl.tar.gz`\ + +* Download the pretrained_dga_model_dsdl.ipynb Jupyter notebook from `https://github.com/splunk/security_content/notebooks`\ + +* Login to the Jupyter Lab for pretrained_dga_model_dsdl container. This container should be listed on Containers page for DSDL app.\ + +* Below steps need to be followed inside Jupyter lab \ + +* Upload the pretrained_dga_model_dsdl.tar.gz file into `app/model/data` path using the upload option in the jupyter notebook.\ + +* Untar the artifact `pretrained_dga_model_dsdl.tar.gz` using `tar -xf app/model/data/pretrained_dga_model_dsdl.tar.gz -C app/model/data`\ + +* Upload `pretrained_dga_model_dsdl.pynb` into Jupyter lab notebooks folder using the upload option in Jupyter lab\ + +* Save the notebook using the save option in jupyter notebook.\ + +* Upload `pretrained_dga_model_dsdl.json` into `notebooks/data` folder.' known_false_positives: False positives may be present if domain name is similar to dga generated domains. references: - https://attack.mitre.org/techniques/T1568/002/ @@ -54,7 +62,7 @@ tags: - DE.CM observable: - name: domain - type: URL Domain + type: URL role: - Attacker - name: src diff --git a/detections/experimental/network/detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl.yml b/detections/experimental/network/detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl.yml new file mode 100644 index 0000000000..1380562fee --- /dev/null +++ b/detections/experimental/network/detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl.yml @@ -0,0 +1,116 @@ +name: Detect suspicious DNS TXT records using pretrained model in DSDL +id: 92f65c3a-968c-11ed-a1eb-0242ac120002 +version: 1 +date: "2023-01-15" +author: Abhinav Mishra, Kumar Sharad and Namratha Sreekanta, Splunk +type: Anomaly +datamodel: +- Network_Resolution +description: The following analytic uses a pre trained deep learning model to + detect suspicious DNS TXT records. The model is trained independently and is + then made available for download. The DNS TXT records are categorized into + commonly identified types like email, verification, http using regular + expressions https://www.tide-project.nl/blog/wtmc2020/. The TXT records that + do not match regular expressions for well known types are labeled as 1 for + "unknown/suspicious" and otherwise 0 for "not suspicious". The deep learning + model we have developed uses DNS TXT responses to analyze patterns of character + sequences to predict if a DNS TXT is suspicious or not. The higher the + pred_is_unknown_proba, the more likely the DNS TXT record is suspicious. + The threshold for flagging a domain as suspicious is set at 0.5. +search: '| tstats `security_content_summariesonly` count min(_time) as firstTime + max(_time) as lastTime from datamodel=Network_Resolution where + DNS.message_type=response AND DNS.record_type=TXT by DNS.src DNS.dest + DNS.answer DNS.record_type + | `drop_dm_object_name("DNS")` + | rename answer as text + | fields firstTime, lastTime, message_type,record_type,src,dest, text + | apply detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl + | rename pred_is_unknown_proba as is_suspicious_score + | where is_suspicious_score > 0.5 + | `security_content_ctime(firstTime)` + | `security_content_ctime(lastTime)` + | rename src as "Source IP", dest as "Destination IP", text as "DNS Answer", record_type as "DNS Record Type" + | table "Source IP","Destination IP","DNS Answer","DNS Record Type", firstTime, lastTime,is_suspicious_score + | `detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl_filter`' + +how_to_implement: 'Steps to deploy detect suspicious DNS TXT records model into Splunk App + DSDL. This detection depends on the Splunk app for Data Science and Deep + Learning which can be found here - https://splunkbase.splunk.com/app/4607/ + and the Network Resolution datamodel which can be found here - + https://splunkbase.splunk.com/app/1621/. The detection uses a pre-trained + deep learning model that needs to be deployed in DSDL app. Follow the steps + for deployment here - `https://github.com/splunk/security_content/wiki/How-to-deploy-pre-trained-Deep-Learning-models-for-ESCU`.\ + + * Download the `artifacts .tar.gz` file from the link - TODO update link here= + Download the `detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl.ipynb` + Jupyter notebook from https://github.com/splunk/security_content/notebooks\ + + * Login to the Jupyter Lab assigned for + detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl container. + This container should be listed on Containers page for DSDL app.\ + + * Below steps need to be followed inside Jupyter lab \ + + * Upload the detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl.tar.gz file + into `app/model/data` path using the upload option in the jupyter notebook.\ + + * Untar the artifact detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl.tar.gz using `tar -xf app/model/data/detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl.tar.gz -C app/model/data`\ + + * Upload detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl.pynb into Jupyter lab notebooks folder using the upload option in Jupyter lab \ + + * Save the notebook using the save option in jupyter notebook. \ + + * Upload `detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl.json` into `notebooks/data` folder.' +known_false_positives: False positives may be present if DNS TXT record contents + are similar to benign DNS TXT record contents. +references: +- https://attack.mitre.org/techniques/T1071/004/ +- https://unit42.paloaltonetworks.com/dns-tunneling-how-dns-can-be-abused-by-malicious-actors/ +- https://en.wikipedia.org/wiki/TXT_record +tags: + analytic_story: + - DNS Hijacking + - Suspicious DNS Traffic + - Command and Control + asset_type: Endpoint + cis20: + - CIS 8 + - CIS 12 + - CIS 13 + confidence: 90 + context: + - Source:Endpoint + - Stage:Command And Control + impact: 50 + kill_chain_phases: + - Command & Control + message: A suspicious DNS TXT response was detected on host $src$ , kindly review. + mitre_attack_id: + - T1568.002 + nist: + - PR.DS + - PR.PT + - DE.AE + - DE.CM + observable: + - name: answer + type: Other + role: + - Attacker + - name: src + type: Hostname + role: + - Victim + product: + - Splunk Enterprise + - Splunk Enterprise Security + - Splunk Cloud + required_fields: + - _time + - DNS.message_type + - DNS.record_type + - DNS.src + - DNS.dest + - DNS.answer + risk_score: 45 + security_domain: network diff --git a/lookups/__mlspl_detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl.mlmodel b/lookups/__mlspl_detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl.mlmodel new file mode 100644 index 0000000000..5b3968aaba --- /dev/null +++ b/lookups/__mlspl_detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl.mlmodel @@ -0,0 +1,2 @@ +algo,model,options +MLTKContainer,"{""__mlspl_type"": [""mltkc.MLTKContainer"", ""MLTKContainer""], ""dict"": {""endpoint_url"": ""https://localhost:54270"", ""out_params"": {""params"": {""mode"": ""stage"", ""algo"": ""detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl""}, ""args"": [""is_unknown"", ""text""], ""target_variable"": [""is_unknown""], ""feature_variables"": [""text""], ""model_name"": ""detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl"", ""algo_name"": ""MLTKContainer"", ""mlspl_limits"": {""handle_new_cat"": ""default"", ""max_distinct_cat_values"": ""100"", ""max_distinct_cat_values_for_classifiers"": ""100"", ""max_distinct_cat_values_for_scoring"": ""100"", ""max_fit_time"": ""600"", ""max_inputs"": ""100000"", ""max_memory_usage_mb"": ""4000"", ""max_model_size_mb"": ""30"", ""max_score_time"": ""600"", ""use_sampling"": ""true""}, ""kfold_cv"": null}, ""feature_variables"": [""text""], ""target_variable"": ""is_unknown""}}","{""params"": {""mode"": ""stage"", ""algo"": ""detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl""}, ""args"": [""is_unknown"", ""text""], ""target_variable"": [""is_unknown""], ""feature_variables"": [""text""], ""model_name"": ""detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl"", ""algo_name"": ""MLTKContainer"", ""mlspl_limits"": {""handle_new_cat"": ""default"", ""max_distinct_cat_values"": ""100"", ""max_distinct_cat_values_for_classifiers"": ""100"", ""max_distinct_cat_values_for_scoring"": ""100"", ""max_fit_time"": ""600"", ""max_inputs"": ""100000"", ""max_memory_usage_mb"": ""4000"", ""max_model_size_mb"": ""30"", ""max_score_time"": ""600"", ""use_sampling"": ""true""}, ""kfold_cv"": null}" diff --git a/lookups/__mlspl_detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl.yml b/lookups/__mlspl_detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl.yml new file mode 100644 index 0000000000..6d1d90cfd5 --- /dev/null +++ b/lookups/__mlspl_detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl.yml @@ -0,0 +1,4 @@ +description: Detect suspicious DNS txt records using Pretrained Model in DSDL +filename: __mlspl_detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl.mlmodel +name: __mlspl_detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl +case_sensitive_match: 'false' \ No newline at end of file diff --git a/notebooks/detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl.ipynb b/notebooks/detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl.ipynb new file mode 100644 index 0000000000..c6f011a8b5 --- /dev/null +++ b/notebooks/detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl.ipynb @@ -0,0 +1,772 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Deep Learning model to detect suspicious DNS TXT records " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This notebook uses a pre-trained deep learning model to predict whether a DNS TXT record is suspicious or not." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "While DNS TXT records initially were supposed to only hold descriptive text, its use and format have been subject to changes and debates over the years. This makes TXT records ideal candidates for malicious and unintended practices which include adding large records to create more efficient DNS amplification attacks, or creating a command and control channel for malware. \n", + "\n", + "The model is trained independently and is then made available for download. The DNS TXT records are categorized into commonly identified types like email, verification, http using regular expressions https://www.tide-project.nl/blog/wtmc2020/. The TXT records that do not match regular expressions for well known types are labeled as 1 for \"unknown/suspicious\" and otherwise 0 for \"not suspicious\". The deep learning model we have developed uses DNX TXT responses to analyze patters of character sequences to predict of a DNS TXT is suspicious or not. Higher the pred_is_unknown_proba, the more likely is the DNS TXT record is suspicious. The threshold for flagging a domain as suspicious is set at 0.5." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Stage 0 - import libraries\n", + "At stage 0 we define all imports necessary to run our subsequent code depending on various libraries." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": { + "deletable": false, + "name": "mltkc_import" + }, + "outputs": [], + "source": [ + "# this definition exposes all python module imports that should be available in all subsequent commands\n", + "import numpy as np \n", + "import pandas as pd \n", + "import os\n", + "import torch\n", + "import torch.nn as nn\n", + "import torch.nn.functional as F\n", + "import torch.optim as optim\n", + "from torch.utils.data.sampler import SubsetRandomSampler\n", + "from torch.utils.data import Dataset\n", + "from torch.utils.data import DataLoader\n", + "from sklearn.feature_extraction.text import CountVectorizer\n", + "from torch.autograd import Variable\n", + "from torch.optim import lr_scheduler\n", + "from collections import Counter,OrderedDict\n", + "import pickle\n", + "\n", + "# global constants\n", + "MODEL_DIRECTORY = \"/srv/app/model/data/\"\n", + "vocab_size = 10002\n", + "embedding_dim = 64\n", + "hidden_size = 64\n", + "fc_hidden_size = 64\n", + "num_output_nodes = 1\n", + "\n", + "dropout = 0.5" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "numpy version: 1.19.2\n", + "pandas version: 1.1.3\n", + "torch version: 1.11.0\n" + ] + } + ], + "source": [ + "# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes\n", + "print(\"numpy version: \" + np.__version__)\n", + "print(\"pandas version: \" + pd.__version__)\n", + "print(\"torch version: \" + torch.__version__)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Stage 1 - get a data sample from Splunk\n", + "There are currently 2 ways to retrieve data from Splunk: Option 1 is to interactively pull data from Splunk into the DLTK Jupyter Lab environment. This is useful when the Splunk REST API is accessible from the Jupyter environment and a valid Splunk auth token is defined in the DLTK app. This option has advantages to quickly experiment with different Splunk SPL queries and further interactively work with the search results in Jupyter.\n", + "\n", + "### Option 1 - pull data from Splunk" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [], + "source": [ + "import libs.SplunkSearch as SplunkSearch" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [ + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "8b33153fe3ca43febb2257b1791064fa", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "VBox(children=(HBox(children=(Textarea(value='| makeresults count=10 \\n| streamstats c as i \\n| eval s = i%3 \\…" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "search = SplunkSearch.SplunkSearch()" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
_timefeature_0feature_1feature_2is
02022-05-18T10:30:54.000+02:000.84603281458839770.320270133670419450.892129479441791811
12022-05-18T10:30:54.000+02:000.32287732744589450.92708458798006180.618015415966510822
22022-05-18T10:30:54.000+02:000.96529306098818780.61272895662114020.695967188570648430
32022-05-18T10:30:54.000+02:000.99857543222606180.72128275176510210.626508353278040941
42022-05-18T10:30:54.000+02:000.49843348795548080.48838610295206310.814449761994183152
52022-05-18T10:30:54.000+02:000.93350205756723880.73463975451886650.0304786232300102760
62022-05-18T10:30:54.000+02:000.83745603915303950.91590728377923370.0180886425077915271
72022-05-18T10:30:54.000+02:000.83552740421146150.38424163172021510.191881095990538682
82022-05-18T10:30:54.000+02:000.354251644574105740.118458491284400220.0149903125129640190
92022-05-18T10:30:54.000+02:000.389599948190152650.70692224986851220.930438622366637101
\n", + "
" + ], + "text/plain": [ + " _time feature_0 feature_1 \\\n", + "0 2022-05-18T10:30:54.000+02:00 0.8460328145883977 0.32027013367041945 \n", + "1 2022-05-18T10:30:54.000+02:00 0.3228773274458945 0.9270845879800618 \n", + "2 2022-05-18T10:30:54.000+02:00 0.9652930609881878 0.6127289566211402 \n", + "3 2022-05-18T10:30:54.000+02:00 0.9985754322260618 0.7212827517651021 \n", + "4 2022-05-18T10:30:54.000+02:00 0.4984334879554808 0.4883861029520631 \n", + "5 2022-05-18T10:30:54.000+02:00 0.9335020575672388 0.7346397545188665 \n", + "6 2022-05-18T10:30:54.000+02:00 0.8374560391530395 0.9159072837792337 \n", + "7 2022-05-18T10:30:54.000+02:00 0.8355274042114615 0.3842416317202151 \n", + "8 2022-05-18T10:30:54.000+02:00 0.35425164457410574 0.11845849128440022 \n", + "9 2022-05-18T10:30:54.000+02:00 0.38959994819015265 0.7069222498685122 \n", + "\n", + " feature_2 i s \n", + "0 0.8921294794417918 1 1 \n", + "1 0.6180154159665108 2 2 \n", + "2 0.6959671885706484 3 0 \n", + "3 0.6265083532780409 4 1 \n", + "4 0.8144497619941831 5 2 \n", + "5 0.03047862323001027 6 0 \n", + "6 0.01808864250779152 7 1 \n", + "7 0.1918810959905386 8 2 \n", + "8 0.01499031251296401 9 0 \n", + "9 0.930438622366637 10 1 " + ] + }, + "execution_count": 5, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "df = search.as_df()\n", + "df" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Option 2 - push data from Splunk\n", + "In Splunk run a search to pipe a dataset into your notebook environment. You utilize the `mode=stage` flag in the in the `| fit` command to do this. The search results are accessible then as csv file with the same model name that is defined in the `into app:` part of the fit statement. Additionally, meta data is retrieved and accessible as json file. In the same way you can further work with the meta data object as it is exposed in the fit and apply function definitions below in stage 3 and 4." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "| makeresults count=10
\n", + "| streamstats c as i
\n", + "| fit MLTKContainer mode=stage algo=barebone_template i into app:barebone_template" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "After you run this search your data set sample is available as a csv inside the container to develop your model. The name is taken from the into keyword (\"barebone_model\" in the example above) or set to \"default\" if no into keyword is present. This step is intended to work with a subset of your data to create your custom model." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "deletable": false, + "name": "mltkc_stage" + }, + "outputs": [], + "source": [ + "# this cell is not executed from MLTK and should only be used for staging data into the notebook environment\n", + "def stage(name):\n", + " with open(\"data/\"+name+\".csv\", 'r') as f:\n", + " df = pd.read_csv(f)\n", + " with open(\"data/\"+name+\".json\", 'r') as f:\n", + " param = json.load(f)\n", + " return df, param" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes\n", + "df, param = stage(\"barebone_template\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "df" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "param" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Stage 2 - create and initialize a model" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": { + "deletable": false, + "name": "mltkc_init" + }, + "outputs": [], + "source": [ + "# initialize your model\n", + "# available inputs: data and parameters\n", + "# returns the model object which will be used as a reference to call fit, apply and summary subsequently\n", + "\n", + "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') \n", + "v = CountVectorizer(analyzer='char_wb', ngram_range=(2, 4), lowercase=True)\n", + "analyzer = v.build_analyzer()\n", + "def tokenize(s):\n", + " tokens = analyzer(s)\n", + " \n", + " try:\n", + " tokens.remove(' ')\n", + " except ValueError:\n", + " pass\n", + " tokens = list(map(str.strip, tokens))\n", + " return tokens\n", + "\n", + "class DataFrameDataset(Dataset):\n", + "\n", + " def __init__(self, df):\n", + " self.texts = []\n", + " self.labels = []\n", + " for i, row in df.iterrows():\n", + " label = row.is_unknown\n", + " text = row.text\n", + " self.texts.append(text)\n", + " self.labels.append(label)\n", + "\n", + " def __len__(self):\n", + " return len(self.labels)\n", + "\n", + " def __getitem__(self, idx):\n", + " return self.texts[idx],self.labels[idx]\n", + "\n", + "\n", + "class DNSTxtClassifier(nn.Module):\n", + " def __init__(self, vocab_size, embed_dim, hidden_size, fc_hidden_size,dropout=0.5,layers=1):\n", + " super().__init__()\n", + " self.embedding = nn.Embedding(vocab_size,embed_dim,padding_idx=0)\n", + " self.lstm = nn.LSTM(embed_dim, hidden_size,batch_first=True,dropout=dropout, num_layers=1)\n", + " self.fc2 = nn.Linear(hidden_size, 1)\n", + " self.sigmoid = nn.Sigmoid()\n", + "#lengths.to(device)\n", + " def forward(self, text, lengths):\n", + " out = self.embedding(text)\n", + " out = nn.utils.rnn.pack_padded_sequence(out,lengths.detach().numpy() , enforce_sorted=False, batch_first=True )\n", + " out, (hidden, cell) = self.lstm(out)\n", + " out = hidden[-1, :, :]\n", + " out = self.fc2(out)\n", + " out = self.sigmoid(out)\n", + " return out\n", + " \n", + "\n", + "\n", + "def init(df,param):\n", + " model = DNSTxtClassifier(vocab_size, embedding_dim, hidden_size,fc_hidden_size,dropout = dropout)\n", + " model.load_state_dict(torch.load(MODEL_DIRECTORY+'/detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl.pt',map_location=torch.device('cpu')))\n", + " model = model.to(device)\n", + " model.eval()\n", + " return model" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "scrolled": true + }, + "outputs": [], + "source": [ + "# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes\n", + "model = init(df,param)\n", + "print(model)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Stage 3 - fit the model" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "deletable": false, + "name": "mltkc_fit" + }, + "outputs": [], + "source": [ + "# train your model\n", + "# returns a fit info json object and may modify the model object\n", + "def fit(model,df,param):\n", + " # model.fit()\n", + " info = {\"message\": \"model trained\"}\n", + " return info" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes\n", + "print(fit(model,df,param))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Stage 4 - apply the model" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "deletable": false, + "name": "mltkc_apply" + }, + "outputs": [], + "source": [ + "# apply your model\n", + "# returns the calculated results\n", + "# apply your model\n", + "# returns the calculated results\n", + "vocab = pickle.load(open(MODEL_DIRECTORY+\"vocab.pkl\", 'rb'))\n", + "\n", + "def remove_spaces_key_value(text):\n", + " return text.replace(' = ','=').replace(\" =\",\"=\").replace('= ','=').replace(' - ','-').replace(\" -\",\"-\").replace('- ','-').replace(' : ',\":\").replace(': ' ,\":\").replace(' :',':')\n", + "\n", + "def get_token_index(text):\n", + " idxs = []\n", + " tokens = tokenize(text)\n", + " for token in tokens:\n", + " if token not in vocab:\n", + " idxs.append(vocab[''])\n", + " else:\n", + " idxs.append(vocab[token])\n", + " return idxs\n", + "\n", + "def preprocess_text(text):\n", + " text = remove_spaces_key_value(text)\n", + " text = text.replace('\"','').replace(\"'\",\"\").lower()\n", + " return text\n", + "\n", + "\n", + "def apply(model,df,param):\n", + " is_unknown_probability_score = []\n", + " is_unknown = []\n", + " for idx, row in df.iterrows():\n", + " text = row['text']\n", + " text = preprocess_text(text)\n", + " indexed_text = get_token_index(text)\n", + " length = [len(indexed_text)]\n", + " token_tensor = torch.LongTensor(indexed_text).to(device) \n", + " token_tensor = token_tensor.unsqueeze(1).T \n", + " length_tensor = torch.LongTensor(length) \n", + " pred_proba = model(token_tensor, length_tensor)[:, 0].item()\n", + " pred = int(pred_proba >= 0.5)\n", + " is_unknown_probability_score.append(pred_proba)\n", + " is_unknown.append(pred)\n", + " output = pd.DataFrame()\n", + " output['pred_is_unknown_proba'] = is_unknown_probability_score\n", + " output['pred_is_unknown'] = is_unknown\n", + " return output" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes\n", + "print(apply(model,df,param))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Stage 5 - save the model" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "deletable": false, + "name": "mltkc_save" + }, + "outputs": [], + "source": [ + "# save model to name in expected convention \"_\"\n", + "def save(model,name):\n", + " with open(MODEL_DIRECTORY + name + \".json\", 'w') as file:\n", + " json.dump(model, file)\n", + " return model" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Stage 6 - load the model" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "deletable": false, + "name": "mltkc_load" + }, + "outputs": [], + "source": [ + "# load model from name in expected convention \"_\"\n", + "def load(name):\n", + " model = DNSTxtClassifier(vocab_size, embedding_dim, hidden_size,fc_hidden_size,dropout = dropout)\n", + " model.load_state_dict(torch.load(MODEL_DIRECTORY+'/detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl.pt',map_location=torch.device('cpu')))\n", + " model = model.to(device)\n", + " model.eval()\n", + " return model " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Stage 7 - provide a summary of the model" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "deletable": false, + "name": "mltkc_summary" + }, + "outputs": [], + "source": [ + "# return a model summary\n", + "def summary(model=None):\n", + " returns = {\"version\": {\"numpy\": np.__version__, \"pandas\": pd.__version__} }\n", + " return returns" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "After implementing your fit, apply, save and load you can train your model:
\n", + "| makeresults count=10
\n", + "| streamstats c as i
\n", + "| eval s = i%3
\n", + "| eval feature_{s}=0
\n", + "| foreach feature_* [eval <<FIELD>>=random()/pow(2,31)]
\n", + "| fit MLTKContainer algo=barebone s from feature_* into app:barebone_model
" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Or apply your model:
\n", + "| makeresults count=10
\n", + "| streamstats c as i
\n", + "| eval s = i%3
\n", + "| eval feature_{s}=0
\n", + "| foreach feature_* [eval <<FIELD>>=random()/pow(2,31)]
\n", + "| apply barebone_model as the_meaning_of_life" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Send data back to Splunk HEC\n", + "When you configured the Splunk HEC Settings in the DLTK app you can easily send back data to an index with [Splunk's HTTP Event Collector (HEC)](https://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector). Read more about data formats and options in the [documentation](https://docs.splunk.com/Documentation/Splunk/latest/Data/FormateventsforHTTPEventCollector#Event_metadata).\n", + "\n", + "### Use cases\n", + "- you want to offload longer running, possibly distributed computations that need to deliver results asynchroneously back into Splunk. \n", + "- you might not want to present results back into the search pipeline after your `| fit` or `| apply` command. \n", + "- you can easily utilize this approach for any logging purposes or other profiling tasks in your ML code so you can actively monitor and analyze your processes.\n", + "\n", + "### Example" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import libs.SplunkHEC as SplunkHEC\n", + "hec = SplunkHEC.SplunkHEC()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# example to send 10 hello world events\n", + "response = hec.send_hello_world(10)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "print(\"HEC endpoint %s \\nreturned with status code %s \\nand response message: %s\" % (response.url, response.status_code, response.text))" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# example to send a JSON object, e.g. to log some data\n", + "from datetime import datetime\n", + "response = hec.send({'event': {'message': 'operation done', 'log_level': 'INFO' }, 'time': datetime.now().timestamp()})" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "print(\"HEC endpoint %s \\nreturned with status code %s \\nand response message: %s\" % (response.url, response.status_code, response.text))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## End of Stages\n", + "All subsequent cells are not tagged and can be used for further freeform code" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.8.5" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} diff --git a/notebooks/detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl.json b/notebooks/detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl.json new file mode 100644 index 0000000000..f32fa38aa8 --- /dev/null +++ b/notebooks/detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl.json @@ -0,0 +1 @@ +{"options": {"params": {"mode": "stage", "algo": "detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl"}, "args": ["is_unknown", "text"], "target_variable": ["is_unknown"], "feature_variables": ["text"], "model_name": "detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl", "algo_name": "MLTKContainer", "mlspl_limits": {"handle_new_cat": "default", "max_distinct_cat_values": "100", "max_distinct_cat_values_for_classifiers": "100", "max_distinct_cat_values_for_scoring": "100", "max_fit_time": "600", "max_inputs": "100000", "max_memory_usage_mb": "4000", "max_model_size_mb": "30", "max_score_time": "600", "use_sampling": "true"}, "kfold_cv": null}, "feature_variables": ["text"], "target_variables": ["is_unknown"]} \ No newline at end of file diff --git a/notebooks/detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl.yml b/notebooks/detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl.yml new file mode 100644 index 0000000000..144b832492 --- /dev/null +++ b/notebooks/detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl.yml @@ -0,0 +1,3 @@ +description: A Jupyter notebook for detecting suspicious DNS TXT records using a pre-trained model in DSDL +filename: detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl.ipynb +name: detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl \ No newline at end of file