From d1788686d575b2f2fd5101e3fe2627d08af931f2 Mon Sep 17 00:00:00 2001 From: Namratha Sreekanta Date: Thu, 15 Sep 2022 17:50:36 -0700 Subject: [PATCH 01/24] initial commit for dga --- .../network/detect_dga_domains_using_dltk.yml | 54 +++ lookups/__mlspl_dga_model_dltk.mlmodel | 2 + lookups/__mlspl_dga_model_dltk.yml | 4 + notebooks/pretrained_dga_detection.ipynb | 427 ++++++++++++++++++ notebooks/pretrained_dga_detection.yml | 3 + 5 files changed, 490 insertions(+) create mode 100644 detections/experimental/network/detect_dga_domains_using_dltk.yml create mode 100644 lookups/__mlspl_dga_model_dltk.mlmodel create mode 100644 lookups/__mlspl_dga_model_dltk.yml create mode 100644 notebooks/pretrained_dga_detection.ipynb create mode 100644 notebooks/pretrained_dga_detection.yml diff --git a/detections/experimental/network/detect_dga_domains_using_dltk.yml b/detections/experimental/network/detect_dga_domains_using_dltk.yml new file mode 100644 index 0000000000..a419cc4e38 --- /dev/null +++ b/detections/experimental/network/detect_dga_domains_using_dltk.yml @@ -0,0 +1,54 @@ +name: Detect DGA domains using Pretrained Deep Learning Model in DLTK +id: b4aefb5f-1037-410d-a149-1e091288ba33 +version: 1 +date: '2022-09-15' +author: Abhinav Mishra, Namratha Sreekanta and Kumar Sharad, Splunk +type: Anomaly +datamodel: +- Splunk_Audit +description: 'The following analytic uses a pretrained machine learning text classifier to detect potentially risky commands. The model is trained independently and then the model file is packaged within ESCU for usage. A command is deemed risky based on the presence of certain trigger keywords, along with the context and the role of the user (please see references). The model uses custom features to predict whether a SPL is risky using text classification. The model takes as input the command text, user and search type and outputs a risk score between [0,1]. A high score indicates higher likelihood of a command being risky. This model is on-prem only.' +search: "| tstats `security_content_summariesonly` values(DNS.answer) as IPs min(_time) as firstTime from datamodel=Network_Resolution by DNS.src, DNS.query +| `drop_dm_object_name("DNS")` | rename query AS domain | fields domain | apply dga_model_dltk as pred | where pred_dga_proba >0.5 | table domain, pred" +how_to_implement: This detection depends on the MLTK app which can be found here - https://splunkbase.splunk.com/app/2890/ and the Splunk Audit datamodel which can be found here - https://splunkbase.splunk.com/app/1621/. Additionally, you need to be ingesting logs which include Search_Activity.search, Search_Activity.user, Search_Activity.search_type from your endpoints. The risk score threshold should be adjusted based on the environment. The detection uses a custom MLTK model hence we need a few more steps for deployment, as outlined here - https://gist.github.com/ksharad-splunk/be2a62227966049047f5e5c4f2adcabb. +known_false_positives: False positives may be present if suspicious behavior is observed, as determined by frequent usage of risky keywords. +references: +- https://docs.splunk.com/Documentation/Splunk/latest/Security/SPLsafeguards#Commands_that_trigger_the_warning +tags: + analytic_story: + - Splunk Vulnerabilities + asset_type: Web Server + cis20: + - CIS 3 + - CIS 6 + confidence: 40 + cve: + - CVE-2022-32154 + context: + - Source:Endpoint + dataset: + - https://github.com/splunk/attack_data/raw/master/datasets/attack_techniques/T1203/search_activity.txt + impact: 50 + kill_chain_phases: + - Actions on Objectives + message: A potentially risky Splunk command has been run by $user$, kindly review. + mitre_attack_id: + - T1059 + nist: + - DE.AE + observable: + - name: user + type: User + role: + - Victim + product: + - Splunk Enterprise + - Splunk Enterprise Security + - Splunk Cloud + required_fields: + - _time + - Search_Activity.search + - Search_Activity.user + - Search_Activity.search_type + risk_score: 20 + security_domain: audit + \ No newline at end of file diff --git a/lookups/__mlspl_dga_model_dltk.mlmodel b/lookups/__mlspl_dga_model_dltk.mlmodel new file mode 100644 index 0000000000..9f9c65d6b9 --- /dev/null +++ b/lookups/__mlspl_dga_model_dltk.mlmodel @@ -0,0 +1,2 @@ +algo,model,options +MLTKContainer,"{""__mlspl_type"": [""mltkc.MLTKContainer"", ""MLTKContainer""], ""dict"": {""endpoint_url"": ""https://localhost:58784"", ""out_params"": {""params"": {""mode"": ""stage"", ""algo"": ""pretrained_dga_detection""}, ""args"": [""is_dga"", ""domain""], ""target_variable"": [""is_dga""], ""feature_variables"": [""domain""], ""model_name"": ""dga_model_dltk"", ""algo_name"": ""MLTKContainer"", ""mlspl_limits"": {""handle_new_cat"": ""default"", ""max_distinct_cat_values"": ""100"", ""max_distinct_cat_values_for_classifiers"": ""100"", ""max_distinct_cat_values_for_scoring"": ""100"", ""max_fit_time"": ""6000"", ""max_inputs"": ""1000000"", ""max_memory_usage_mb"": ""4000"", ""max_model_size_mb"": ""30"", ""max_score_time"": ""6000"", ""use_sampling"": ""true""}, ""kfold_cv"": null}, ""feature_variables"": [""domain""], ""target_variable"": ""is_dga""}}","{""params"": {""mode"": ""stage"", ""algo"": ""pretrained_dga_detection""}, ""args"": [""is_dga"", ""domain""], ""target_variable"": [""is_dga""], ""feature_variables"": [""domain""], ""model_name"": ""dga_model_dltk"", ""algo_name"": ""MLTKContainer"", ""mlspl_limits"": {""handle_new_cat"": ""default"", ""max_distinct_cat_values"": ""100"", ""max_distinct_cat_values_for_classifiers"": ""100"", ""max_distinct_cat_values_for_scoring"": ""100"", ""max_fit_time"": ""6000"", ""max_inputs"": ""1000000"", ""max_memory_usage_mb"": ""4000"", ""max_model_size_mb"": ""30"", ""max_score_time"": ""6000"", ""use_sampling"": ""true""}, ""kfold_cv"": null}" diff --git a/lookups/__mlspl_dga_model_dltk.yml b/lookups/__mlspl_dga_model_dltk.yml new file mode 100644 index 0000000000..ee32c8a0b7 --- /dev/null +++ b/lookups/__mlspl_dga_model_dltk.yml @@ -0,0 +1,4 @@ +description: Detect DGA domains using Pretrained Model in DLTK +filename: __mlspl_dga_model_dltk.mlmodel +name: __mlspl_dga_model_dltk +case_sensitive_match: 'false' \ No newline at end of file diff --git a/notebooks/pretrained_dga_detection.ipynb b/notebooks/pretrained_dga_detection.ipynb new file mode 100644 index 0000000000..5cbfecac67 --- /dev/null +++ b/notebooks/pretrained_dga_detection.ipynb @@ -0,0 +1,427 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Deep Learning Toolkit for Splunk - Barebone Notebook" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This notebook contains a barebone example workflow how to work on custom containerized code that seamlessly interfaces with the Deep Learning Toolkit for Splunk." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Note: By default every time you save this notebook the cells are exported into a python module which is then invoked by Splunk MLTK commands like | fit ... | apply ... | summary . Please read the Model Development Guide in the Deep Learning Toolkit app for more information." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Stage 0 - import libraries\n", + "At stage 0 we define all imports necessary to run our subsequent code depending on various libraries." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "deletable": false, + "name": "mltkc_import" + }, + "outputs": [], + "source": [ + "import time\n", + "from sklearn.metrics import classification_report, confusion_matrix\n", + "import pandas as pd\n", + "import tensorflow as tf\n", + "from tensorflow.keras import Input \n", + "from tensorflow.keras.models import Sequential\n", + "from tensorflow.keras import layers\n", + "from tensorflow.keras.layers import Embedding\n", + "from tensorflow.keras.preprocessing.text import Tokenizer\n", + "from tensorflow.keras.preprocessing.sequence import pad_sequences\n", + "from tensorflow.keras.layers import Embedding,LSTM,Dropout,Dense,Activation\n", + "from tensorflow.keras.activations import sigmoid\n", + "import matplotlib.pyplot as plt\n", + "import math\n", + "import numpy as np\n", + "import pickle\n", + "from sklearn.utils import shuffle\n", + "from tensorflow.keras.preprocessing import sequence\n", + "import datetime\n", + "from collections import Counter\n", + "MODEL_DIRECTORY = \"/srv/app/model/data/\"\n", + "MAX_VOCAB = 40\n", + "pd.options.mode.chained_assignment = None \n", + "import warnings\n", + "warnings.filterwarnings('ignore')" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes\n", + "print(\"numpy version: \" + np.__version__)\n", + "print(\"pandas version: \" + pd.__version__)\n", + "print(\"tensorflow version: \" + tf.__version__)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Stage 1 - get a data sample from Splunk\n", + "In Splunk run a search to pipe a dataset into your notebook environment. Note: mode=stage is used in the | fit command to do this." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "| makeresults count=10
\n", + "| streamstats c as i
\n", + "| eval s = i%3
\n", + "| eval feature_{s}=0
\n", + "| foreach feature_* [eval <<FIELD>>=random()/pow(2,31)]
\n", + "| fit MLTKContainer mode=stage algo=barebone epochs=10 batch_size=1 s from feature_* into app:barebone_model" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "After you run this search your data set sample is available as a csv inside the container to develop your model. The name is taken from the into keyword (\"barebone_model\" in the example above) or set to \"default\" if no into keyword is present. This step is intended to work with a subset of your data to create your custom model." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "deletable": false, + "name": "mltkc_stage" + }, + "outputs": [], + "source": [ + "# this cell is not executed from MLTK and should only be used for staging data into the notebook environment\n", + "def stage(name):\n", + " with open(\"data/\"+name+\".csv\", 'r') as f:\n", + " df = pd.read_csv(f)\n", + " with open(\"data/\"+name+\".json\", 'r') as f:\n", + " param = json.load(f)\n", + " return df, param" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes\n", + "df, param = stage(\"dga_model_dltk\")\n", + "print(df.describe())\n", + "print(param)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Stage 2 - create and initialize a model" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "deletable": false, + "name": "mltkc_init" + }, + "outputs": [], + "source": [ + "# initialize your model\n", + "# available inputs: data and parameters\n", + "# returns the model object which will be used as a reference to call fit, apply and summary subsequently\n", + "def init(df,param):\n", + " model = tf.keras.models.load_model(MODEL_DIRECTORY + \"pretrained_dga_apply_dga_model_dltk\")\n", + " return model" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "scrolled": true + }, + "outputs": [], + "source": [ + "# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes\n", + "print(init(df,param))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Stage 3 - fit the model" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "deletable": false, + "name": "mltkc_fit" + }, + "outputs": [], + "source": [ + "# train your model\n", + "# returns a fit info json object and may modify the model object\n", + "def fit(model,df,param):\n", + " # model.fit()\n", + " info = {\"message\": \"model trained\"}\n", + " return info" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes\n", + "print(fit(model,df,param))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Stage 4 - apply the model" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "deletable": false, + "name": "mltkc_apply" + }, + "outputs": [], + "source": [ + "# apply your model\n", + "# returns the calculated results\n", + "tokenizer = pickle.load(open(MODEL_DIRECTORY+\"pretrained_dga_apply_dga_model_dltk/tokenizer\", 'rb'))\n", + "vectorizer_words = pickle.load(open(MODEL_DIRECTORY+\"pretrained_dga_apply_dga_model_dltk/vectorizer_words\", 'rb'))\n", + "vectorizer_non_dga_domains = pickle.load(open(MODEL_DIRECTORY+\"pretrained_dga_apply_dga_model_dltk/vectorizer_non_dga_domains\", 'rb'))\n", + "weight_words = pickle.load(open(MODEL_DIRECTORY+\"pretrained_dga_apply_dga_model_dltk/weight_words\", 'rb'))\n", + "weight_non_dga_grams = pickle.load(open(MODEL_DIRECTORY+\"pretrained_dga_apply_dga_model_dltk/weight_non_dga_grams\", 'rb'))\n", + "domains = pickle.load(open(MODEL_DIRECTORY+\"pretrained_dga_apply_dga_model_dltk/domains\", 'rb'))\n", + "model = tf.keras.models.load_model(MODEL_DIRECTORY + \"pretrained_dga_apply_dga_model_dltk\")\n", + "\n", + "def entropy(domain):\n", + " p, lns = Counter(domain), float(len(domain))\n", + " return -sum( count/lns * math.log(count/lns, 2) for count in p.values())\n", + " \n", + "alexa_domains = domains['domain']\n", + "\n", + " \n", + "def is_in_alexa1m(domain,domains):\n", + " test_in_alexa_domains = set(alexa_domains) & domains\n", + " return ((domain in test_in_alexa_domains))\n", + "\n", + " \n", + " \n", + "def add_features(df):\n", + " print (\"1. Done adding ngram features\")\n", + " X_1= weight_words * vectorizer_words.transform(df['domain']).T \n", + " X_2= weight_non_dga_grams * vectorizer_non_dga_domains.transform(df['domain']).T\n", + " X_3 = df['domain'].map(lambda x: entropy(x)) \n", + " print (\"2. Done adding entropy\")\n", + " X_4 = df['domain'].map(lambda x: len(x))\n", + " print (\"3. Done adding length of domain\")\n", + " domains = set(df['domain'])\n", + " X_5 = df['domain'].map(lambda x: is_in_alexa1m(x,domains))\n", + " print (\"4. Done adding domain present in alexa domains\")\n", + " X_5 = X_5.astype(int)\n", + " input2 = np.c_[X_1,X_2,X_3,X_4,X_5] #\n", + " print (\"appending done\")\n", + " return input2 \n", + "\n", + "def prep_text(texts):\n", + " text_sequences = tokenizer.texts_to_sequences(texts)\n", + " return sequence.pad_sequences(text_sequences, maxlen=MAX_VOCAB)\n", + "\n", + "def apply(model,df,param):\n", + " input1 = prep_text(df['domain'])\n", + " input2 = add_features(df)\n", + " \n", + " yhat = model.predict([input1,input2])\n", + " y_ = (yhat > 0.5).astype(\"int32\")\n", + " output = pd.DataFrame()\n", + " #output['domain'] = df['domain']\n", + " #output['true_dga'] = df['is_dga'] \n", + " output['pred_dga']=y_.ravel()\n", + " output['pred_dga_proba']=yhat.ravel()\n", + " return output" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes\n", + "print(apply(model,df,param))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Stage 5 - save the model" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "deletable": false, + "name": "mltkc_save" + }, + "outputs": [], + "source": [ + "# save model to name in expected convention \"_\"\n", + "def save(model,name):\n", + " with open(MODEL_DIRECTORY + name + \".json\", 'w') as file:\n", + " json.dump(model, file)\n", + " return model" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Stage 6 - load the model" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "deletable": false, + "name": "mltkc_load" + }, + "outputs": [], + "source": [ + "# load model from name in expected convention \"_\"\n", + "def load(name):\n", + " model = tf.keras.models.load_model(MODEL_DIRECTORY + \"pretrained_dga_apply_dga_model_dltk\")\n", + " return model" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Stage 7 - provide a summary of the model" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "deletable": false, + "name": "mltkc_summary" + }, + "outputs": [], + "source": [ + "# return a model summary\n", + "def summary(model=None):\n", + " returns = {\"version\": {\"numpy\": np.__version__, \"pandas\": pd.__version__} }\n", + " return returns" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "After implementing your fit, apply, save and load you can train your model:
\n", + "| makeresults count=10
\n", + "| streamstats c as i
\n", + "| eval s = i%3
\n", + "| eval feature_{s}=0
\n", + "| foreach feature_* [eval <<FIELD>>=random()/pow(2,31)]
\n", + "| fit MLTKContainer algo=barebone s from feature_* into app:barebone_model
" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Or apply your model:
\n", + "| makeresults count=10
\n", + "| streamstats c as i
\n", + "| eval s = i%3
\n", + "| eval feature_{s}=0
\n", + "| foreach feature_* [eval <<FIELD>>=random()/pow(2,31)]
\n", + "| apply barebone_model as the_meaning_of_life" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## End of Stages\n", + "All subsequent cells are not tagged and can be used for further freeform code" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.8.5" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} diff --git a/notebooks/pretrained_dga_detection.yml b/notebooks/pretrained_dga_detection.yml new file mode 100644 index 0000000000..abe821a40c --- /dev/null +++ b/notebooks/pretrained_dga_detection.yml @@ -0,0 +1,3 @@ +description: A Jupyter notebook for detecting DGA domains using DLTK +filename: pretrained_dga_detection.ipynb +name: pretrained_dga_detection \ No newline at end of file From c417a9a4af1f35d3805c9905bf22d18f82e213e8 Mon Sep 17 00:00:00 2001 From: Namratha Sreekanta Date: Fri, 16 Sep 2022 15:13:58 -0700 Subject: [PATCH 02/24] changing the search in the yml --- .../network/detect_dga_domains_using_dltk.yml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/detections/experimental/network/detect_dga_domains_using_dltk.yml b/detections/experimental/network/detect_dga_domains_using_dltk.yml index a419cc4e38..0e9ac0d583 100644 --- a/detections/experimental/network/detect_dga_domains_using_dltk.yml +++ b/detections/experimental/network/detect_dga_domains_using_dltk.yml @@ -6,9 +6,9 @@ author: Abhinav Mishra, Namratha Sreekanta and Kumar Sharad, Splunk type: Anomaly datamodel: - Splunk_Audit -description: 'The following analytic uses a pretrained machine learning text classifier to detect potentially risky commands. The model is trained independently and then the model file is packaged within ESCU for usage. A command is deemed risky based on the presence of certain trigger keywords, along with the context and the role of the user (please see references). The model uses custom features to predict whether a SPL is risky using text classification. The model takes as input the command text, user and search type and outputs a risk score between [0,1]. A high score indicates higher likelihood of a command being risky. This model is on-prem only.' +description: 'The following analytic uses a pretrained deep learning classifier to detect DGA domains. The model is trained independently and the model file is available in AWS S3. A domain is classified as DGA generated when the domain name consists of unusual character sequences or concatenated english words. The model uses the domain name to anlayze patterns of character sequences along with custom features to classify a domain as DGA or not. The model takes a domain name consisting of second-level and top-level domain names as input and outputs a pred_is_dga, either a 0 or 1. If pred_is_dga is 1 then the domain name is DGA generated else it is non DGA.' search: "| tstats `security_content_summariesonly` values(DNS.answer) as IPs min(_time) as firstTime from datamodel=Network_Resolution by DNS.src, DNS.query -| `drop_dm_object_name("DNS")` | rename query AS domain | fields domain | apply dga_model_dltk as pred | where pred_dga_proba >0.5 | table domain, pred" +| `drop_dm_object_name("DNS")` | rename query AS domain | fields domain | apply dga_model_dltk | where predicted_is_dga=1 | table domain, pred" how_to_implement: This detection depends on the MLTK app which can be found here - https://splunkbase.splunk.com/app/2890/ and the Splunk Audit datamodel which can be found here - https://splunkbase.splunk.com/app/1621/. Additionally, you need to be ingesting logs which include Search_Activity.search, Search_Activity.user, Search_Activity.search_type from your endpoints. The risk score threshold should be adjusted based on the environment. The detection uses a custom MLTK model hence we need a few more steps for deployment, as outlined here - https://gist.github.com/ksharad-splunk/be2a62227966049047f5e5c4f2adcabb. known_false_positives: False positives may be present if suspicious behavior is observed, as determined by frequent usage of risky keywords. references: @@ -24,13 +24,13 @@ tags: cve: - CVE-2022-32154 context: - - Source:Endpoint + - Source:Network_Resolution dataset: - https://github.com/splunk/attack_data/raw/master/datasets/attack_techniques/T1203/search_activity.txt impact: 50 kill_chain_phases: - Actions on Objectives - message: A potentially risky Splunk command has been run by $user$, kindly review. + message: A potentially a DGA domain $domain$ was detected, kindly review. mitre_attack_id: - T1059 nist: From 24cbcc1af21777e74ea183bc6ed3b0362c88c9f9 Mon Sep 17 00:00:00 2001 From: Namratha Sreekanta Date: Fri, 16 Sep 2022 17:32:18 -0700 Subject: [PATCH 03/24] changing notebook code and adding json file for the model, moved all contents under dga_dltk folder --- notebooks/dga_dltk/dga_model_dltk.json | 1 + .../pretrained_dga_detection.ipynb | 80 +++++-------------- .../pretrained_dga_detection.yml | 0 3 files changed, 23 insertions(+), 58 deletions(-) create mode 100644 notebooks/dga_dltk/dga_model_dltk.json rename notebooks/{ => dga_dltk}/pretrained_dga_detection.ipynb (78%) rename notebooks/{ => dga_dltk}/pretrained_dga_detection.yml (100%) diff --git a/notebooks/dga_dltk/dga_model_dltk.json b/notebooks/dga_dltk/dga_model_dltk.json new file mode 100644 index 0000000000..5f97da2a03 --- /dev/null +++ b/notebooks/dga_dltk/dga_model_dltk.json @@ -0,0 +1 @@ +{"options": {"params": {"mode": "stage", "algo": "pretrained_dga_detection"}, "args": ["is_dga", "domain"], "target_variable": ["is_dga"], "feature_variables": ["domain"], "model_name": "dga_model_dltk", "algo_name": "MLTKContainer", "mlspl_limits": {"handle_new_cat": "default", "max_distinct_cat_values": "100", "max_distinct_cat_values_for_classifiers": "100", "max_distinct_cat_values_for_scoring": "100", "max_fit_time": "6000", "max_inputs": "1000000", "max_memory_usage_mb": "4000", "max_model_size_mb": "30", "max_score_time": "6000", "use_sampling": "true"}, "kfold_cv": null}, "feature_variables": ["domain"], "target_variables": ["is_dga"]} \ No newline at end of file diff --git a/notebooks/pretrained_dga_detection.ipynb b/notebooks/dga_dltk/pretrained_dga_detection.ipynb similarity index 78% rename from notebooks/pretrained_dga_detection.ipynb rename to notebooks/dga_dltk/pretrained_dga_detection.ipynb index 5cbfecac67..161b682b3a 100644 --- a/notebooks/pretrained_dga_detection.ipynb +++ b/notebooks/dga_dltk/pretrained_dga_detection.ipynb @@ -4,21 +4,30 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# Deep Learning Toolkit for Splunk - Barebone Notebook" + "# Deep Learning model to detect DGA domains" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "This notebook contains a barebone example workflow how to work on custom containerized code that seamlessly interfaces with the Deep Learning Toolkit for Splunk." + "This notebook uses a pre-trained deep learning model to predict whether a domain is DGA generated or not." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "Note: By default every time you save this notebook the cells are exported into a python module which is then invoked by Splunk MLTK commands like | fit ... | apply ... | summary . Please read the Model Development Guide in the Deep Learning Toolkit app for more information." + "### What are DGA domains ?" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Adversaries may make use of Domain Generation Algorithms (DGAs) to dynamically identify a destination domain for command and control traffic rather than relying on a list of static IP addresses or domains. This has the advantage of making it much harder for defenders to block, track, or take over the command and control channel, as there potentially could be thousands of domains that malware can check for instructions.\n", + "\n", + "DGAs can take the form of apparently random or \"gibberish\" strings (ex: istgmxdejdnxuyla.ru) when they construct domain names by generating each letter. Alternatively, some DGAs employ whole words as the unit by concatenating words together instead of letters (ex: cityjulydish.net). Many DGAs are time-based, generating a different domain for each time period (hourly, daily, monthly, etc). Others incorporate a seed value as well to make predicting future domains more difficult for defenders https://attack.mitre.org/techniques/T1568/002/" ] }, { @@ -82,26 +91,7 @@ "metadata": {}, "source": [ "## Stage 1 - get a data sample from Splunk\n", - "In Splunk run a search to pipe a dataset into your notebook environment. Note: mode=stage is used in the | fit command to do this." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "| makeresults count=10
\n", - "| streamstats c as i
\n", - "| eval s = i%3
\n", - "| eval feature_{s}=0
\n", - "| foreach feature_* [eval <<FIELD>>=random()/pow(2,31)]
\n", - "| fit MLTKContainer mode=stage algo=barebone epochs=10 batch_size=1 s from feature_* into app:barebone_model" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "After you run this search your data set sample is available as a csv inside the container to develop your model. The name is taken from the into keyword (\"barebone_model\" in the example above) or set to \"default\" if no into keyword is present. This step is intended to work with a subset of your data to create your custom model." + "This notebook does not train" ] }, { @@ -154,7 +144,7 @@ "# available inputs: data and parameters\n", "# returns the model object which will be used as a reference to call fit, apply and summary subsequently\n", "def init(df,param):\n", - " model = tf.keras.models.load_model(MODEL_DIRECTORY + \"pretrained_dga_apply_dga_model_dltk\")\n", + " model = tf.keras.models.load_model(MODEL_DIRECTORY + \"pretrained_dga_detection_dga_model_dltk\")\n", " return model" ] }, @@ -222,13 +212,13 @@ "source": [ "# apply your model\n", "# returns the calculated results\n", - "tokenizer = pickle.load(open(MODEL_DIRECTORY+\"pretrained_dga_apply_dga_model_dltk/tokenizer\", 'rb'))\n", - "vectorizer_words = pickle.load(open(MODEL_DIRECTORY+\"pretrained_dga_apply_dga_model_dltk/vectorizer_words\", 'rb'))\n", - "vectorizer_non_dga_domains = pickle.load(open(MODEL_DIRECTORY+\"pretrained_dga_apply_dga_model_dltk/vectorizer_non_dga_domains\", 'rb'))\n", - "weight_words = pickle.load(open(MODEL_DIRECTORY+\"pretrained_dga_apply_dga_model_dltk/weight_words\", 'rb'))\n", - "weight_non_dga_grams = pickle.load(open(MODEL_DIRECTORY+\"pretrained_dga_apply_dga_model_dltk/weight_non_dga_grams\", 'rb'))\n", - "domains = pickle.load(open(MODEL_DIRECTORY+\"pretrained_dga_apply_dga_model_dltk/domains\", 'rb'))\n", - "model = tf.keras.models.load_model(MODEL_DIRECTORY + \"pretrained_dga_apply_dga_model_dltk\")\n", + "tokenizer = pickle.load(open(MODEL_DIRECTORY+\"pretrained_dga_detection_dga_model_dltk/tokenizer\", 'rb'))\n", + "vectorizer_words = pickle.load(open(MODEL_DIRECTORY+\"pretrained_dga_detection_dga_model_dltk/vectorizer_words\", 'rb'))\n", + "vectorizer_non_dga_domains = pickle.load(open(MODEL_DIRECTORY+\"pretrained_dga_detection_dga_model_dltk/vectorizer_non_dga_domains\", 'rb'))\n", + "weight_words = pickle.load(open(MODEL_DIRECTORY+\"pretrained_dga_detection_dga_model_dltk/weight_words\", 'rb'))\n", + "weight_non_dga_grams = pickle.load(open(MODEL_DIRECTORY+\"pretrained_dga_detection_dga_model_dltk/weight_non_dga_grams\", 'rb'))\n", + "domains = pickle.load(open(MODEL_DIRECTORY+\"pretrained_dga_detection_dga_model_dltk/domains\", 'rb'))\n", + "model = tf.keras.models.load_model(MODEL_DIRECTORY + \"pretrained_dga_detection_dga_model_dltk\")\n", "\n", "def entropy(domain):\n", " p, lns = Counter(domain), float(len(domain))\n", @@ -328,7 +318,7 @@ "source": [ "# load model from name in expected convention \"_\"\n", "def load(name):\n", - " model = tf.keras.models.load_model(MODEL_DIRECTORY + \"pretrained_dga_apply_dga_model_dltk\")\n", + " model = tf.keras.models.load_model(MODEL_DIRECTORY + \"pretrained_dga_detection_dga_model_dltk\")\n", " return model" ] }, @@ -354,32 +344,6 @@ " return returns" ] }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "After implementing your fit, apply, save and load you can train your model:
\n", - "| makeresults count=10
\n", - "| streamstats c as i
\n", - "| eval s = i%3
\n", - "| eval feature_{s}=0
\n", - "| foreach feature_* [eval <<FIELD>>=random()/pow(2,31)]
\n", - "| fit MLTKContainer algo=barebone s from feature_* into app:barebone_model
" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Or apply your model:
\n", - "| makeresults count=10
\n", - "| streamstats c as i
\n", - "| eval s = i%3
\n", - "| eval feature_{s}=0
\n", - "| foreach feature_* [eval <<FIELD>>=random()/pow(2,31)]
\n", - "| apply barebone_model as the_meaning_of_life" - ] - }, { "cell_type": "markdown", "metadata": {}, diff --git a/notebooks/pretrained_dga_detection.yml b/notebooks/dga_dltk/pretrained_dga_detection.yml similarity index 100% rename from notebooks/pretrained_dga_detection.yml rename to notebooks/dga_dltk/pretrained_dga_detection.yml From 43c213a74c06a163b84a4e429577e6713f2110f1 Mon Sep 17 00:00:00 2001 From: Namratha Sreekanta Date: Tue, 20 Sep 2022 11:20:13 -0700 Subject: [PATCH 04/24] modifying the search and adding a README file --- .../network/detect_dga_domains_using_dltk.yml | 18 ++++----- notebooks/dga_dltk/README.md | 38 +++++++++++++++++++ 2 files changed, 46 insertions(+), 10 deletions(-) create mode 100644 notebooks/dga_dltk/README.md diff --git a/detections/experimental/network/detect_dga_domains_using_dltk.yml b/detections/experimental/network/detect_dga_domains_using_dltk.yml index 0e9ac0d583..d30d4eefba 100644 --- a/detections/experimental/network/detect_dga_domains_using_dltk.yml +++ b/detections/experimental/network/detect_dga_domains_using_dltk.yml @@ -2,14 +2,14 @@ name: Detect DGA domains using Pretrained Deep Learning Model in DLTK id: b4aefb5f-1037-410d-a149-1e091288ba33 version: 1 date: '2022-09-15' -author: Abhinav Mishra, Namratha Sreekanta and Kumar Sharad, Splunk +author: Abhinav Mishra, Kumar Sharad and Namratha Sreekanta, Splunk type: Anomaly datamodel: -- Splunk_Audit +- Network_Resolution description: 'The following analytic uses a pretrained deep learning classifier to detect DGA domains. The model is trained independently and the model file is available in AWS S3. A domain is classified as DGA generated when the domain name consists of unusual character sequences or concatenated english words. The model uses the domain name to anlayze patterns of character sequences along with custom features to classify a domain as DGA or not. The model takes a domain name consisting of second-level and top-level domain names as input and outputs a pred_is_dga, either a 0 or 1. If pred_is_dga is 1 then the domain name is DGA generated else it is non DGA.' search: "| tstats `security_content_summariesonly` values(DNS.answer) as IPs min(_time) as firstTime from datamodel=Network_Resolution by DNS.src, DNS.query -| `drop_dm_object_name("DNS")` | rename query AS domain | fields domain | apply dga_model_dltk | where predicted_is_dga=1 | table domain, pred" -how_to_implement: This detection depends on the MLTK app which can be found here - https://splunkbase.splunk.com/app/2890/ and the Splunk Audit datamodel which can be found here - https://splunkbase.splunk.com/app/1621/. Additionally, you need to be ingesting logs which include Search_Activity.search, Search_Activity.user, Search_Activity.search_type from your endpoints. The risk score threshold should be adjusted based on the environment. The detection uses a custom MLTK model hence we need a few more steps for deployment, as outlined here - https://gist.github.com/ksharad-splunk/be2a62227966049047f5e5c4f2adcabb. +| `drop_dm_object_name("DNS")` | rename query AS domain | fields domain | apply dga_model_dltk | rename pred_dga_proba AS dga_score | where dga_score>0.5 | table domain, dga_score" +how_to_implement: This detection depends on the DSDL app which can be found here - https://splunkbase.splunk.com/app/4607/ and the Network Resolution datamodel which can be found here - https://splunkbase.splunk.com/app/1621/. The dga_score threshold is set at 0.5 and this should be adjusted based on the environment. The detection uses a pre-trained deep learning model that needs to be deployed in DLTK. Steps for deployment are outlined here - ** update gist here **. known_false_positives: False positives may be present if suspicious behavior is observed, as determined by frequent usage of risky keywords. references: - https://docs.splunk.com/Documentation/Splunk/latest/Security/SPLsafeguards#Commands_that_trigger_the_warning @@ -36,8 +36,8 @@ tags: nist: - DE.AE observable: - - name: user - type: User + - name: src + type: src role: - Victim product: @@ -46,9 +46,7 @@ tags: - Splunk Cloud required_fields: - _time - - Search_Activity.search - - Search_Activity.user - - Search_Activity.search_type + - DNS.query risk_score: 20 - security_domain: audit + security_domain: Network_Resolution \ No newline at end of file diff --git a/notebooks/dga_dltk/README.md b/notebooks/dga_dltk/README.md new file mode 100644 index 0000000000..dc91de7eb7 --- /dev/null +++ b/notebooks/dga_dltk/README.md @@ -0,0 +1,38 @@ +Deploying DGA model in Splunk App for Data Science and Deep Learning (DSDL) +=========================================================================== + +### Set up of the DSDL app + + +1. Start the docker daemon +2. Install the DSDL app on Splunk instance and follow the steps outlined in the Overview > User Guide (Overview drop down menu in the DSDL app) +3. Additional information and FAQs are available here https://splunkbase.splunk.com/app/4607/#/details + +### Download the artifacts - notebooks, binaries + +1. Download the artifacts .tar.gz file from S3 bucket https://splunk-seal.s3.us-west-2.amazonaws.com/pretrained_dga_detection_dga_model_dltk.tar.gz + +### Deploy the artifacts + +1. Login into Splunk instance, launch DSDL app. +2. Select Containers and it should list all the containers. +3. Select the Golden image 3.9 and type of cluster - docker/kubernetes and start the dev container +4. Wait for the container to start up and populate urls for the container +5. Login into the __dev__ container jupyter lab url ex: https://:8888/lab? (Password: Splunk4DeepLearning) +6. Open a terminal on Jupyterlab and execute the following commands +* Upload the pretrained_dga_detection_dga_model_dltk.tar.gz file into app/model/data using the upload option in the jupyter notebook +* Untar the artifact pretrained_dga_detection_dga_model_dltk.tar.gz +``` +tar -xf app/model/data/pretrained_dga_detection_dga_model_dltk.tar.gz -C app/model/data +``` +* Upload notebook pretrained_dga_detection.ipynb into notebooks/ using the upload in Jupyter notebook and save the notebook using the save option in jupyter notebook +* Upload dga_model_dltk.json into notebooks/data folder + + 7. The .mlmlmodel is an essential file that contains information about the pretrained model. To make the pre-trained model available, the model spec files need to be placed in the lookup of mltk-container under etc/apps. + * Place __mlspl_dga_model_dltk.mlmodel into install_path_of_splunk/etc/apps/mltk-container/lookups + + 8. Restart splunk server + 9. Relaunch the DSDL app, select the Containers menu. Select Golden image, suitable environment - either Docker or Kubernetes and select the specific container with the name and Start the container. + +### Applying pretrained model into DSDL +Run the search 'Detect DGA domains using Pretrained Deep Learning Model in DLTK' and search should run successfully without errors. From 8fa7fffa2a54d3aa8435e2ca968e5b5a02ebb974 Mon Sep 17 00:00:00 2001 From: Namratha Sreekanta Date: Tue, 20 Sep 2022 11:33:06 -0700 Subject: [PATCH 05/24] changing the desc to match the detection --- .../experimental/network/detect_dga_domains_using_dltk.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/detections/experimental/network/detect_dga_domains_using_dltk.yml b/detections/experimental/network/detect_dga_domains_using_dltk.yml index d30d4eefba..b65a51b973 100644 --- a/detections/experimental/network/detect_dga_domains_using_dltk.yml +++ b/detections/experimental/network/detect_dga_domains_using_dltk.yml @@ -6,7 +6,7 @@ author: Abhinav Mishra, Kumar Sharad and Namratha Sreekanta, Splunk type: Anomaly datamodel: - Network_Resolution -description: 'The following analytic uses a pretrained deep learning classifier to detect DGA domains. The model is trained independently and the model file is available in AWS S3. A domain is classified as DGA generated when the domain name consists of unusual character sequences or concatenated english words. The model uses the domain name to anlayze patterns of character sequences along with custom features to classify a domain as DGA or not. The model takes a domain name consisting of second-level and top-level domain names as input and outputs a pred_is_dga, either a 0 or 1. If pred_is_dga is 1 then the domain name is DGA generated else it is non DGA.' +description: 'The following analytic uses a pretrained deep learning classifier to detect DGA domains. The model is trained independently and the model file is available in AWS S3. A domain is classified as DGA generated when the domain name consists of unusual character sequences or concatenated english words. The model uses the domain name to anlayze patterns of character sequences along with custom features to classify a domain as DGA or not. The model takes a domain name consisting of second-level and top-level domain names as input and outputs a dga_score. Higher the dga_score, the more likely the input domain is a dga domain. The threshold for flagging a domain as dga is set at 0.5.' search: "| tstats `security_content_summariesonly` values(DNS.answer) as IPs min(_time) as firstTime from datamodel=Network_Resolution by DNS.src, DNS.query | `drop_dm_object_name("DNS")` | rename query AS domain | fields domain | apply dga_model_dltk | rename pred_dga_proba AS dga_score | where dga_score>0.5 | table domain, dga_score" how_to_implement: This detection depends on the DSDL app which can be found here - https://splunkbase.splunk.com/app/4607/ and the Network Resolution datamodel which can be found here - https://splunkbase.splunk.com/app/1621/. The dga_score threshold is set at 0.5 and this should be adjusted based on the environment. The detection uses a pre-trained deep learning model that needs to be deployed in DLTK. Steps for deployment are outlined here - ** update gist here **. From 56aea9de227f97074f322b9bbbad179b013befbb Mon Sep 17 00:00:00 2001 From: Namratha Sreekanta Date: Tue, 20 Sep 2022 12:28:53 -0700 Subject: [PATCH 06/24] Adding in Sharad's inputs on descriptioj --- .../experimental/network/detect_dga_domains_using_dltk.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/detections/experimental/network/detect_dga_domains_using_dltk.yml b/detections/experimental/network/detect_dga_domains_using_dltk.yml index b65a51b973..9528b8dd10 100644 --- a/detections/experimental/network/detect_dga_domains_using_dltk.yml +++ b/detections/experimental/network/detect_dga_domains_using_dltk.yml @@ -6,7 +6,7 @@ author: Abhinav Mishra, Kumar Sharad and Namratha Sreekanta, Splunk type: Anomaly datamodel: - Network_Resolution -description: 'The following analytic uses a pretrained deep learning classifier to detect DGA domains. The model is trained independently and the model file is available in AWS S3. A domain is classified as DGA generated when the domain name consists of unusual character sequences or concatenated english words. The model uses the domain name to anlayze patterns of character sequences along with custom features to classify a domain as DGA or not. The model takes a domain name consisting of second-level and top-level domain names as input and outputs a dga_score. Higher the dga_score, the more likely the input domain is a dga domain. The threshold for flagging a domain as dga is set at 0.5.' +description: 'The following analytic uses a pretrained deep learning model to detect Domain Generation Algorithm (DGA) generated domains. The model is trained independently and then made available in AWS S3. One of the prominent indicators of a domain being DGA generated is if the domain name consists of unusual character sequences or concatenated dictionary words. Adversaries often use clever techniques to obfuscate machine generated domain names as human generated. Predicting DGA generated domain names requires analysis and building a model based on carefully chosen features. The deep learning model we have developed uses the domain name to anlayze patterns of character sequences along with carefully chosen custom features to predict if a domain is DGA generated. The model takes a domain name consisting of second-level and top-level domain names as input and outputs a dga_score. Higher the dga_score, the more likely the input domain is a DGA domain. The threshold for flagging a domain as DGA is set at 0.5.' search: "| tstats `security_content_summariesonly` values(DNS.answer) as IPs min(_time) as firstTime from datamodel=Network_Resolution by DNS.src, DNS.query | `drop_dm_object_name("DNS")` | rename query AS domain | fields domain | apply dga_model_dltk | rename pred_dga_proba AS dga_score | where dga_score>0.5 | table domain, dga_score" how_to_implement: This detection depends on the DSDL app which can be found here - https://splunkbase.splunk.com/app/4607/ and the Network Resolution datamodel which can be found here - https://splunkbase.splunk.com/app/1621/. The dga_score threshold is set at 0.5 and this should be adjusted based on the environment. The detection uses a pre-trained deep learning model that needs to be deployed in DLTK. Steps for deployment are outlined here - ** update gist here **. From f5fb35ec4c0222bf9f06ec391a891f8d745e6bc0 Mon Sep 17 00:00:00 2001 From: Namratha Sreekanta Date: Tue, 20 Sep 2022 16:00:12 -0700 Subject: [PATCH 07/24] changing README and few fields regarding the search --- .../network/detect_dga_domains_using_dltk.yml | 27 ++++++++------ notebooks/dga_dltk/README.md | 36 +++++++++---------- 2 files changed, 33 insertions(+), 30 deletions(-) diff --git a/detections/experimental/network/detect_dga_domains_using_dltk.yml b/detections/experimental/network/detect_dga_domains_using_dltk.yml index 9528b8dd10..d0e3daa61e 100644 --- a/detections/experimental/network/detect_dga_domains_using_dltk.yml +++ b/detections/experimental/network/detect_dga_domains_using_dltk.yml @@ -15,31 +15,38 @@ references: - https://docs.splunk.com/Documentation/Splunk/latest/Security/SPLsafeguards#Commands_that_trigger_the_warning tags: analytic_story: - - Splunk Vulnerabilities + - Data Protection + - Prohibited Traffic Allowed or Protocol Mismatch + - DNS Hijacking + - Suspicious DNS Traffic + - Dynamic DNS + - Command and Control asset_type: Web Server cis20: - - CIS 3 - - CIS 6 + - CIS 8 + - CIS 12 + - CIS 13 confidence: 40 cve: - CVE-2022-32154 context: - - Source:Network_Resolution + - Source:Endpoint + - Stage:Initial Access dataset: - https://github.com/splunk/attack_data/raw/master/datasets/attack_techniques/T1203/search_activity.txt impact: 50 kill_chain_phases: - Actions on Objectives + - Command & Control message: A potentially a DGA domain $domain$ was detected, kindly review. mitre_attack_id: - - T1059 + - T1189 nist: + - PR.DS + - PR.PT - DE.AE + - DE.CM observable: - - name: src - type: src - role: - - Victim product: - Splunk Enterprise - Splunk Enterprise Security @@ -48,5 +55,5 @@ tags: - _time - DNS.query risk_score: 20 - security_domain: Network_Resolution + security_domain: Network \ No newline at end of file diff --git a/notebooks/dga_dltk/README.md b/notebooks/dga_dltk/README.md index dc91de7eb7..15f6902ac8 100644 --- a/notebooks/dga_dltk/README.md +++ b/notebooks/dga_dltk/README.md @@ -8,31 +8,27 @@ Deploying DGA model in Splunk App for Data Science and Deep Learning (DSDL) 2. Install the DSDL app on Splunk instance and follow the steps outlined in the Overview > User Guide (Overview drop down menu in the DSDL app) 3. Additional information and FAQs are available here https://splunkbase.splunk.com/app/4607/#/details -### Download the artifacts - notebooks, binaries +### Download the model artifacts - notebooks, binaries 1. Download the artifacts .tar.gz file from S3 bucket https://splunk-seal.s3.us-west-2.amazonaws.com/pretrained_dga_detection_dga_model_dltk.tar.gz -### Deploy the artifacts +### Deploy the model artifacts 1. Login into Splunk instance, launch DSDL app. 2. Select Containers and it should list all the containers. -3. Select the Golden image 3.9 and type of cluster - docker/kubernetes and start the dev container -4. Wait for the container to start up and populate urls for the container -5. Login into the __dev__ container jupyter lab url ex: https://:8888/lab? (Password: Splunk4DeepLearning) +3. Select Container Image as Golden image 3.9 and Cluster target as per env setup and start the dev container. +4. Wait for the container to start up and urls to populate for the container. +5. Login into the __dev__ container jupyter lab url ex: https://:8888/lab? 6. Open a terminal on Jupyterlab and execute the following commands -* Upload the pretrained_dga_detection_dga_model_dltk.tar.gz file into app/model/data using the upload option in the jupyter notebook -* Untar the artifact pretrained_dga_detection_dga_model_dltk.tar.gz -``` -tar -xf app/model/data/pretrained_dga_detection_dga_model_dltk.tar.gz -C app/model/data -``` -* Upload notebook pretrained_dga_detection.ipynb into notebooks/ using the upload in Jupyter notebook and save the notebook using the save option in jupyter notebook -* Upload dga_model_dltk.json into notebooks/data folder + * Upload the pretrained_dga_detection_dga_model_dltk.tar.gz file into app/model/data using the upload option in the jupyter notebook. + * Untar the artifact pretrained_dga_detection_dga_model_dltk.tar.gz + ``` + tar -xf app/model/data/pretrained_dga_detection_dga_model_dltk.tar.gz -C app/model/data + ``` + * Upload notebook pretrained_dga_detection.ipynb into notebooks folder using the upload option in Jupyter lab and save the notebook using the save option in jupyter notebook. + * Upload dga_model_dltk.json into notebooks/data folder. - 7. The .mlmlmodel is an essential file that contains information about the pretrained model. To make the pre-trained model available, the model spec files need to be placed in the lookup of mltk-container under etc/apps. - * Place __mlspl_dga_model_dltk.mlmodel into install_path_of_splunk/etc/apps/mltk-container/lookups - - 8. Restart splunk server - 9. Relaunch the DSDL app, select the Containers menu. Select Golden image, suitable environment - either Docker or Kubernetes and select the specific container with the name and Start the container. - -### Applying pretrained model into DSDL -Run the search 'Detect DGA domains using Pretrained Deep Learning Model in DLTK' and search should run successfully without errors. + 7. Refresh the DSDL app, select the Containers menu. + 8. Select Container Image as Golden image 3.9 and Cluster target as per env setup and start the new container dga_model_dltk. + 9. Select the container and click Start. + 10. The dga detection model dga_model_dltk is now deployed within DLTK. \ No newline at end of file From 055f0a175afe74a711b92d71e357350ad834b36b Mon Sep 17 00:00:00 2001 From: Namratha Sreekanta Date: Wed, 21 Sep 2022 13:20:47 -0700 Subject: [PATCH 08/24] adding review comments --- ...omains_using_pretrained_model in dsdl.yml} | 49 +++++++++++-------- notebooks/{dga_dltk => dga_dsdl}/README.md | 4 +- .../dga_model_dltk.json | 0 .../pretrained_dga_detection.ipynb | 0 .../pretrained_dga_detection.yml | 2 +- 5 files changed, 31 insertions(+), 24 deletions(-) rename detections/experimental/network/{detect_dga_domains_using_dltk.yml => detect_dga_domains_using_pretrained_model in dsdl.yml} (64%) rename notebooks/{dga_dltk => dga_dsdl}/README.md (98%) rename notebooks/{dga_dltk => dga_dsdl}/dga_model_dltk.json (100%) rename notebooks/{dga_dltk => dga_dsdl}/pretrained_dga_detection.ipynb (100%) rename notebooks/{dga_dltk => dga_dsdl}/pretrained_dga_detection.yml (82%) diff --git a/detections/experimental/network/detect_dga_domains_using_dltk.yml b/detections/experimental/network/detect_dga_domains_using_pretrained_model in dsdl.yml similarity index 64% rename from detections/experimental/network/detect_dga_domains_using_dltk.yml rename to detections/experimental/network/detect_dga_domains_using_pretrained_model in dsdl.yml index d0e3daa61e..820f3e02e4 100644 --- a/detections/experimental/network/detect_dga_domains_using_dltk.yml +++ b/detections/experimental/network/detect_dga_domains_using_pretrained_model in dsdl.yml @@ -1,5 +1,5 @@ -name: Detect DGA domains using Pretrained Deep Learning Model in DLTK -id: b4aefb5f-1037-410d-a149-1e091288ba33 +name: Detect DGA domains using pretrained model in DSDL +id: 92e24f32-9b9a-4060-bba2-2a0eb3bf3493 version: 1 date: '2022-09-15' author: Abhinav Mishra, Kumar Sharad and Namratha Sreekanta, Splunk @@ -7,12 +7,14 @@ type: Anomaly datamodel: - Network_Resolution description: 'The following analytic uses a pretrained deep learning model to detect Domain Generation Algorithm (DGA) generated domains. The model is trained independently and then made available in AWS S3. One of the prominent indicators of a domain being DGA generated is if the domain name consists of unusual character sequences or concatenated dictionary words. Adversaries often use clever techniques to obfuscate machine generated domain names as human generated. Predicting DGA generated domain names requires analysis and building a model based on carefully chosen features. The deep learning model we have developed uses the domain name to anlayze patterns of character sequences along with carefully chosen custom features to predict if a domain is DGA generated. The model takes a domain name consisting of second-level and top-level domain names as input and outputs a dga_score. Higher the dga_score, the more likely the input domain is a DGA domain. The threshold for flagging a domain as DGA is set at 0.5.' -search: "| tstats `security_content_summariesonly` values(DNS.answer) as IPs min(_time) as firstTime from datamodel=Network_Resolution by DNS.src, DNS.query -| `drop_dm_object_name("DNS")` | rename query AS domain | fields domain | apply dga_model_dltk | rename pred_dga_proba AS dga_score | where dga_score>0.5 | table domain, dga_score" -how_to_implement: This detection depends on the DSDL app which can be found here - https://splunkbase.splunk.com/app/4607/ and the Network Resolution datamodel which can be found here - https://splunkbase.splunk.com/app/1621/. The dga_score threshold is set at 0.5 and this should be adjusted based on the environment. The detection uses a pre-trained deep learning model that needs to be deployed in DLTK. Steps for deployment are outlined here - ** update gist here **. -known_false_positives: False positives may be present if suspicious behavior is observed, as determined by frequent usage of risky keywords. +search: '| tstats `security_content_summariesonly` values(DNS.answer) as IPs min(_time) as firstTime max(_time) as lastTime from datamodel=Network_Resolution by DNS.src, DNS.query +| `drop_dm_object_name("DNS")` | rename query AS domain | fields IPs, src, domain, firstTime, lastTime, src | apply dga_model_dltk | rename pred_dga_proba AS dga_score | where dga_score<0.5 | table src, domain, IPs, firstTime, lastTime,dga_score' +how_to_implement: This detection depends on the DSDL app which can be found here - https://splunkbase.splunk.com/app/4607/ and the Network Resolution datamodel which can be found here - https://splunkbase.splunk.com/app/1621/. The dga_score threshold is set at 0.5 and this should be adjusted based on the environment. The detection uses a pre-trained deep learning model that needs to be deployed in DSDL. Steps for deployment are outlined here - https://github.com/splunk/security_content/notebooks/dga_dsdl/README.md. +known_false_positives: False positives may be present if domain name is similar to dga generated domains. references: -- https://docs.splunk.com/Documentation/Splunk/latest/Security/SPLsafeguards#Commands_that_trigger_the_warning +- https://attack.mitre.org/techniques/T1568/002/ +- https://unit42.paloaltonetworks.com/threat-brief-understanding-domain-generation-algorithms-dga/ +- https://en.wikipedia.org/wiki/Domain_generation_algorithm tags: analytic_story: - Data Protection @@ -26,34 +28,39 @@ tags: - CIS 8 - CIS 12 - CIS 13 - confidence: 40 - cve: - - CVE-2022-32154 + confidence: 90 context: - Source:Endpoint - - Stage:Initial Access - dataset: - - https://github.com/splunk/attack_data/raw/master/datasets/attack_techniques/T1203/search_activity.txt - impact: 50 + - Stage:Command and Control + impact: 70 kill_chain_phases: - - Actions on Objectives - Command & Control message: A potentially a DGA domain $domain$ was detected, kindly review. mitre_attack_id: - - T1189 + - T1568.002 nist: - PR.DS - PR.PT - DE.AE - DE.CM observable: + - name: domain + type: Other + role: + - Attacker + - name: src + type: Hostname + role: + - Victim product: - Splunk Enterprise - Splunk Enterprise Security - Splunk Cloud required_fields: - - _time - - DNS.query - risk_score: 20 - security_domain: Network - \ No newline at end of file + - IPs + - src + - domain + - firstTime + - lastTime + risk_score: 63 + security_domain: Network \ No newline at end of file diff --git a/notebooks/dga_dltk/README.md b/notebooks/dga_dsdl/README.md similarity index 98% rename from notebooks/dga_dltk/README.md rename to notebooks/dga_dsdl/README.md index 15f6902ac8..5fc322e331 100644 --- a/notebooks/dga_dltk/README.md +++ b/notebooks/dga_dsdl/README.md @@ -18,7 +18,7 @@ Deploying DGA model in Splunk App for Data Science and Deep Learning (DSDL) 2. Select Containers and it should list all the containers. 3. Select Container Image as Golden image 3.9 and Cluster target as per env setup and start the dev container. 4. Wait for the container to start up and urls to populate for the container. -5. Login into the __dev__ container jupyter lab url ex: https://:8888/lab? +5. Login into the __dev__ container jupyter lab url ex: https://{container_url}:8888/lab? 6. Open a terminal on Jupyterlab and execute the following commands * Upload the pretrained_dga_detection_dga_model_dltk.tar.gz file into app/model/data using the upload option in the jupyter notebook. * Untar the artifact pretrained_dga_detection_dga_model_dltk.tar.gz @@ -31,4 +31,4 @@ Deploying DGA model in Splunk App for Data Science and Deep Learning (DSDL) 7. Refresh the DSDL app, select the Containers menu. 8. Select Container Image as Golden image 3.9 and Cluster target as per env setup and start the new container dga_model_dltk. 9. Select the container and click Start. - 10. The dga detection model dga_model_dltk is now deployed within DLTK. \ No newline at end of file + 10. The dga detection model dga_model_dltk is now deployed within DSDL. \ No newline at end of file diff --git a/notebooks/dga_dltk/dga_model_dltk.json b/notebooks/dga_dsdl/dga_model_dltk.json similarity index 100% rename from notebooks/dga_dltk/dga_model_dltk.json rename to notebooks/dga_dsdl/dga_model_dltk.json diff --git a/notebooks/dga_dltk/pretrained_dga_detection.ipynb b/notebooks/dga_dsdl/pretrained_dga_detection.ipynb similarity index 100% rename from notebooks/dga_dltk/pretrained_dga_detection.ipynb rename to notebooks/dga_dsdl/pretrained_dga_detection.ipynb diff --git a/notebooks/dga_dltk/pretrained_dga_detection.yml b/notebooks/dga_dsdl/pretrained_dga_detection.yml similarity index 82% rename from notebooks/dga_dltk/pretrained_dga_detection.yml rename to notebooks/dga_dsdl/pretrained_dga_detection.yml index abe821a40c..7711fc0946 100644 --- a/notebooks/dga_dltk/pretrained_dga_detection.yml +++ b/notebooks/dga_dsdl/pretrained_dga_detection.yml @@ -1,3 +1,3 @@ -description: A Jupyter notebook for detecting DGA domains using DLTK +description: A Jupyter notebook for detecting DGA domains using a pre-trained model in DSDL filename: pretrained_dga_detection.ipynb name: pretrained_dga_detection \ No newline at end of file From 6678a4ee55caa3fec8af87c9fbef5448bb46f51e Mon Sep 17 00:00:00 2001 From: Namratha Sreekanta Date: Wed, 21 Sep 2022 13:35:47 -0700 Subject: [PATCH 09/24] fixing build failure --- ... => detect_dga_domains_using_pretrained_model_in_dsdl.yml} | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) rename detections/experimental/network/{detect_dga_domains_using_pretrained_model in dsdl.yml => detect_dga_domains_using_pretrained_model_in_dsdl.yml} (95%) diff --git a/detections/experimental/network/detect_dga_domains_using_pretrained_model in dsdl.yml b/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml similarity index 95% rename from detections/experimental/network/detect_dga_domains_using_pretrained_model in dsdl.yml rename to detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml index 820f3e02e4..fc3cd3ab95 100644 --- a/detections/experimental/network/detect_dga_domains_using_pretrained_model in dsdl.yml +++ b/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml @@ -8,7 +8,7 @@ datamodel: - Network_Resolution description: 'The following analytic uses a pretrained deep learning model to detect Domain Generation Algorithm (DGA) generated domains. The model is trained independently and then made available in AWS S3. One of the prominent indicators of a domain being DGA generated is if the domain name consists of unusual character sequences or concatenated dictionary words. Adversaries often use clever techniques to obfuscate machine generated domain names as human generated. Predicting DGA generated domain names requires analysis and building a model based on carefully chosen features. The deep learning model we have developed uses the domain name to anlayze patterns of character sequences along with carefully chosen custom features to predict if a domain is DGA generated. The model takes a domain name consisting of second-level and top-level domain names as input and outputs a dga_score. Higher the dga_score, the more likely the input domain is a DGA domain. The threshold for flagging a domain as DGA is set at 0.5.' search: '| tstats `security_content_summariesonly` values(DNS.answer) as IPs min(_time) as firstTime max(_time) as lastTime from datamodel=Network_Resolution by DNS.src, DNS.query -| `drop_dm_object_name("DNS")` | rename query AS domain | fields IPs, src, domain, firstTime, lastTime, src | apply dga_model_dltk | rename pred_dga_proba AS dga_score | where dga_score<0.5 | table src, domain, IPs, firstTime, lastTime,dga_score' +| `drop_dm_object_name("DNS")` | rename query AS domain | fields IPs, src, domain, firstTime, lastTime, src | apply dga_model_dltk | rename pred_dga_proba AS dga_score | where dga_score<0.5 | table src, domain, IPs, firstTime, lastTime,dga_score | `detect_dga_domains_using_pretrained_model_in_dsdl_filter`' how_to_implement: This detection depends on the DSDL app which can be found here - https://splunkbase.splunk.com/app/4607/ and the Network Resolution datamodel which can be found here - https://splunkbase.splunk.com/app/1621/. The dga_score threshold is set at 0.5 and this should be adjusted based on the environment. The detection uses a pre-trained deep learning model that needs to be deployed in DSDL. Steps for deployment are outlined here - https://github.com/splunk/security_content/notebooks/dga_dsdl/README.md. known_false_positives: False positives may be present if domain name is similar to dga generated domains. references: @@ -31,7 +31,7 @@ tags: confidence: 90 context: - Source:Endpoint - - Stage:Command and Control + - Stage:Command And Control impact: 70 kill_chain_phases: - Command & Control From d22e6db5efe588cd65335cc62f830e1ffd5eaf37 Mon Sep 17 00:00:00 2001 From: Namratha Sreekanta Date: Thu, 22 Sep 2022 14:57:42 -0700 Subject: [PATCH 10/24] minor change in the detection --- .../detect_dga_domains_using_pretrained_model_in_dsdl.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml b/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml index fc3cd3ab95..0fd3818ba5 100644 --- a/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml +++ b/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml @@ -8,7 +8,7 @@ datamodel: - Network_Resolution description: 'The following analytic uses a pretrained deep learning model to detect Domain Generation Algorithm (DGA) generated domains. The model is trained independently and then made available in AWS S3. One of the prominent indicators of a domain being DGA generated is if the domain name consists of unusual character sequences or concatenated dictionary words. Adversaries often use clever techniques to obfuscate machine generated domain names as human generated. Predicting DGA generated domain names requires analysis and building a model based on carefully chosen features. The deep learning model we have developed uses the domain name to anlayze patterns of character sequences along with carefully chosen custom features to predict if a domain is DGA generated. The model takes a domain name consisting of second-level and top-level domain names as input and outputs a dga_score. Higher the dga_score, the more likely the input domain is a DGA domain. The threshold for flagging a domain as DGA is set at 0.5.' search: '| tstats `security_content_summariesonly` values(DNS.answer) as IPs min(_time) as firstTime max(_time) as lastTime from datamodel=Network_Resolution by DNS.src, DNS.query -| `drop_dm_object_name("DNS")` | rename query AS domain | fields IPs, src, domain, firstTime, lastTime, src | apply dga_model_dltk | rename pred_dga_proba AS dga_score | where dga_score<0.5 | table src, domain, IPs, firstTime, lastTime,dga_score | `detect_dga_domains_using_pretrained_model_in_dsdl_filter`' +| `drop_dm_object_name("DNS")` | rename query AS domain | fields IPs, src, domain, firstTime, lastTime, src | apply dga_model_dltk | rename pred_dga_proba AS dga_score | where dga_score>0.5 | table src, domain, IPs, firstTime, lastTime,dga_score | `detect_dga_domains_using_pretrained_model_in_dsdl_filter`' how_to_implement: This detection depends on the DSDL app which can be found here - https://splunkbase.splunk.com/app/4607/ and the Network Resolution datamodel which can be found here - https://splunkbase.splunk.com/app/1621/. The dga_score threshold is set at 0.5 and this should be adjusted based on the environment. The detection uses a pre-trained deep learning model that needs to be deployed in DSDL. Steps for deployment are outlined here - https://github.com/splunk/security_content/notebooks/dga_dsdl/README.md. known_false_positives: False positives may be present if domain name is similar to dga generated domains. references: From 83eb61820ebcb115f89e2526e91172f211642a90 Mon Sep 17 00:00:00 2001 From: Namratha Sreekanta Date: Thu, 22 Sep 2022 16:30:31 -0700 Subject: [PATCH 11/24] Changing README.md instructions --- notebooks/dga_dsdl/README.md | 33 +++++++++++++++------------------ 1 file changed, 15 insertions(+), 18 deletions(-) diff --git a/notebooks/dga_dsdl/README.md b/notebooks/dga_dsdl/README.md index 5fc322e331..d4692c70af 100644 --- a/notebooks/dga_dsdl/README.md +++ b/notebooks/dga_dsdl/README.md @@ -4,9 +4,8 @@ Deploying DGA model in Splunk App for Data Science and Deep Learning (DSDL) ### Set up of the DSDL app -1. Start the docker daemon -2. Install the DSDL app on Splunk instance and follow the steps outlined in the Overview > User Guide (Overview drop down menu in the DSDL app) -3. Additional information and FAQs are available here https://splunkbase.splunk.com/app/4607/#/details +1. Install the DSDL app on Splunk instance and follow the steps outlined in the Overview > User Guide. +2. Additional information and FAQs are available here https://splunkbase.splunk.com/app/4607/#/details ### Download the model artifacts - notebooks, binaries @@ -16,19 +15,17 @@ Deploying DGA model in Splunk App for Data Science and Deep Learning (DSDL) 1. Login into Splunk instance, launch DSDL app. 2. Select Containers and it should list all the containers. -3. Select Container Image as Golden image 3.9 and Cluster target as per env setup and start the dev container. +3. Select Container Image as Golden image 3.9 and Cluster target as per env setup and start the dga_model_dltk container. 4. Wait for the container to start up and urls to populate for the container. -5. Login into the __dev__ container jupyter lab url ex: https://{container_url}:8888/lab? -6. Open a terminal on Jupyterlab and execute the following commands - * Upload the pretrained_dga_detection_dga_model_dltk.tar.gz file into app/model/data using the upload option in the jupyter notebook. - * Untar the artifact pretrained_dga_detection_dga_model_dltk.tar.gz - ``` - tar -xf app/model/data/pretrained_dga_detection_dga_model_dltk.tar.gz -C app/model/data - ``` - * Upload notebook pretrained_dga_detection.ipynb into notebooks folder using the upload option in Jupyter lab and save the notebook using the save option in jupyter notebook. - * Upload dga_model_dltk.json into notebooks/data folder. - - 7. Refresh the DSDL app, select the Containers menu. - 8. Select Container Image as Golden image 3.9 and Cluster target as per env setup and start the new container dga_model_dltk. - 9. Select the container and click Start. - 10. The dga detection model dga_model_dltk is now deployed within DSDL. \ No newline at end of file +5. Login into the dga_model_dltk container Jupyter lab url ex: https://{container_url}:port_num/lab? + *Use the password provided in the Overview > User Guide of DSDL app +6. The below steps are performed within the Jupyter Lab. + * Upload the pretrained_dga_detection_dga_model_dltk.tar.gz file into app/model/data path using the upload option in the jupyter notebook. + * Open a terminal on Jupyterlab and execute the following commands + * Untar the artifact pretrained_dga_detection_dga_model_dltk.tar.gz + ``` + tar -xf app/model/data/pretrained_dga_detection_dga_model_dltk.tar.gz -C app/model/data + ``` + * Upload {confirm_path}notebook pretrained_dga_detection.ipynb into jupyter lab notebooks folder using the upload option in Jupyter lab and save the notebook using the save option in jupyter notebook. + * Upload dga_model_dltk.json into notebooks/data folder. + 7. The dga detection model dga_model_dltk is now deployed within DSDL. \ No newline at end of file From 34b6e086385dab77188f3b752b1e1488e8cee303 Mon Sep 17 00:00:00 2001 From: Namratha Sreekanta Date: Thu, 22 Sep 2022 16:32:17 -0700 Subject: [PATCH 12/24] Changing README.md instructions --- notebooks/dga_dsdl/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/notebooks/dga_dsdl/README.md b/notebooks/dga_dsdl/README.md index d4692c70af..bbcf9d9733 100644 --- a/notebooks/dga_dsdl/README.md +++ b/notebooks/dga_dsdl/README.md @@ -18,7 +18,7 @@ Deploying DGA model in Splunk App for Data Science and Deep Learning (DSDL) 3. Select Container Image as Golden image 3.9 and Cluster target as per env setup and start the dga_model_dltk container. 4. Wait for the container to start up and urls to populate for the container. 5. Login into the dga_model_dltk container Jupyter lab url ex: https://{container_url}:port_num/lab? - *Use the password provided in the Overview > User Guide of DSDL app + * Use the password provided in the Overview > User Guide of DSDL app 6. The below steps are performed within the Jupyter Lab. * Upload the pretrained_dga_detection_dga_model_dltk.tar.gz file into app/model/data path using the upload option in the jupyter notebook. * Open a terminal on Jupyterlab and execute the following commands From 2932ebb1fac00b796b3482303b95acd15221e7bb Mon Sep 17 00:00:00 2001 From: Namratha Sreekanta Date: Fri, 23 Sep 2022 16:28:58 -0700 Subject: [PATCH 13/24] Adding more feedback --- ...domains_using_pretrained_model_in_dsdl.yml | 2 +- notebooks/dga_dsdl/README.md | 31 ------------------- notebooks/dga_dsdl/dga_model_dltk.json | 1 - .../pretrained_dga_detection.yml | 4 +-- ....ipynb => pretrained_dga_model_dsdl.ipynb} | 20 ++++++------ notebooks/pretrained_dga_model_dsdl.json | 1 + 6 files changed, 14 insertions(+), 45 deletions(-) delete mode 100644 notebooks/dga_dsdl/README.md delete mode 100644 notebooks/dga_dsdl/dga_model_dltk.json rename notebooks/{dga_dsdl => }/pretrained_dga_detection.yml (55%) rename notebooks/{dga_dsdl/pretrained_dga_detection.ipynb => pretrained_dga_model_dsdl.ipynb} (95%) create mode 100644 notebooks/pretrained_dga_model_dsdl.json diff --git a/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml b/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml index 0fd3818ba5..576e55287a 100644 --- a/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml +++ b/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml @@ -8,7 +8,7 @@ datamodel: - Network_Resolution description: 'The following analytic uses a pretrained deep learning model to detect Domain Generation Algorithm (DGA) generated domains. The model is trained independently and then made available in AWS S3. One of the prominent indicators of a domain being DGA generated is if the domain name consists of unusual character sequences or concatenated dictionary words. Adversaries often use clever techniques to obfuscate machine generated domain names as human generated. Predicting DGA generated domain names requires analysis and building a model based on carefully chosen features. The deep learning model we have developed uses the domain name to anlayze patterns of character sequences along with carefully chosen custom features to predict if a domain is DGA generated. The model takes a domain name consisting of second-level and top-level domain names as input and outputs a dga_score. Higher the dga_score, the more likely the input domain is a DGA domain. The threshold for flagging a domain as DGA is set at 0.5.' search: '| tstats `security_content_summariesonly` values(DNS.answer) as IPs min(_time) as firstTime max(_time) as lastTime from datamodel=Network_Resolution by DNS.src, DNS.query -| `drop_dm_object_name("DNS")` | rename query AS domain | fields IPs, src, domain, firstTime, lastTime, src | apply dga_model_dltk | rename pred_dga_proba AS dga_score | where dga_score>0.5 | table src, domain, IPs, firstTime, lastTime,dga_score | `detect_dga_domains_using_pretrained_model_in_dsdl_filter`' +| `drop_dm_object_name("DNS")` | rename query AS domain | fields IPs, src, domain, firstTime, lastTime, src | apply pretrained_dga_model_dsdl | rename pred_dga_proba AS dga_score | where dga_score>0.5 | `security_content_ctime(firstTime)` | `security_content_ctime(lastTime)` | table src, domain, IPs, firstTime, lastTime, dga_score | `detect_dga_domains_using_pretrained_model_in_dsdl_filter`' how_to_implement: This detection depends on the DSDL app which can be found here - https://splunkbase.splunk.com/app/4607/ and the Network Resolution datamodel which can be found here - https://splunkbase.splunk.com/app/1621/. The dga_score threshold is set at 0.5 and this should be adjusted based on the environment. The detection uses a pre-trained deep learning model that needs to be deployed in DSDL. Steps for deployment are outlined here - https://github.com/splunk/security_content/notebooks/dga_dsdl/README.md. known_false_positives: False positives may be present if domain name is similar to dga generated domains. references: diff --git a/notebooks/dga_dsdl/README.md b/notebooks/dga_dsdl/README.md deleted file mode 100644 index bbcf9d9733..0000000000 --- a/notebooks/dga_dsdl/README.md +++ /dev/null @@ -1,31 +0,0 @@ -Deploying DGA model in Splunk App for Data Science and Deep Learning (DSDL) -=========================================================================== - -### Set up of the DSDL app - - -1. Install the DSDL app on Splunk instance and follow the steps outlined in the Overview > User Guide. -2. Additional information and FAQs are available here https://splunkbase.splunk.com/app/4607/#/details - -### Download the model artifacts - notebooks, binaries - -1. Download the artifacts .tar.gz file from S3 bucket https://splunk-seal.s3.us-west-2.amazonaws.com/pretrained_dga_detection_dga_model_dltk.tar.gz - -### Deploy the model artifacts - -1. Login into Splunk instance, launch DSDL app. -2. Select Containers and it should list all the containers. -3. Select Container Image as Golden image 3.9 and Cluster target as per env setup and start the dga_model_dltk container. -4. Wait for the container to start up and urls to populate for the container. -5. Login into the dga_model_dltk container Jupyter lab url ex: https://{container_url}:port_num/lab? - * Use the password provided in the Overview > User Guide of DSDL app -6. The below steps are performed within the Jupyter Lab. - * Upload the pretrained_dga_detection_dga_model_dltk.tar.gz file into app/model/data path using the upload option in the jupyter notebook. - * Open a terminal on Jupyterlab and execute the following commands - * Untar the artifact pretrained_dga_detection_dga_model_dltk.tar.gz - ``` - tar -xf app/model/data/pretrained_dga_detection_dga_model_dltk.tar.gz -C app/model/data - ``` - * Upload {confirm_path}notebook pretrained_dga_detection.ipynb into jupyter lab notebooks folder using the upload option in Jupyter lab and save the notebook using the save option in jupyter notebook. - * Upload dga_model_dltk.json into notebooks/data folder. - 7. The dga detection model dga_model_dltk is now deployed within DSDL. \ No newline at end of file diff --git a/notebooks/dga_dsdl/dga_model_dltk.json b/notebooks/dga_dsdl/dga_model_dltk.json deleted file mode 100644 index 5f97da2a03..0000000000 --- a/notebooks/dga_dsdl/dga_model_dltk.json +++ /dev/null @@ -1 +0,0 @@ -{"options": {"params": {"mode": "stage", "algo": "pretrained_dga_detection"}, "args": ["is_dga", "domain"], "target_variable": ["is_dga"], "feature_variables": ["domain"], "model_name": "dga_model_dltk", "algo_name": "MLTKContainer", "mlspl_limits": {"handle_new_cat": "default", "max_distinct_cat_values": "100", "max_distinct_cat_values_for_classifiers": "100", "max_distinct_cat_values_for_scoring": "100", "max_fit_time": "6000", "max_inputs": "1000000", "max_memory_usage_mb": "4000", "max_model_size_mb": "30", "max_score_time": "6000", "use_sampling": "true"}, "kfold_cv": null}, "feature_variables": ["domain"], "target_variables": ["is_dga"]} \ No newline at end of file diff --git a/notebooks/dga_dsdl/pretrained_dga_detection.yml b/notebooks/pretrained_dga_detection.yml similarity index 55% rename from notebooks/dga_dsdl/pretrained_dga_detection.yml rename to notebooks/pretrained_dga_detection.yml index 7711fc0946..79979162bf 100644 --- a/notebooks/dga_dsdl/pretrained_dga_detection.yml +++ b/notebooks/pretrained_dga_detection.yml @@ -1,3 +1,3 @@ description: A Jupyter notebook for detecting DGA domains using a pre-trained model in DSDL -filename: pretrained_dga_detection.ipynb -name: pretrained_dga_detection \ No newline at end of file +filename: pretrained_dga_model_dsdl.ipynb +name: pretrained_dga_model_dsdl \ No newline at end of file diff --git a/notebooks/dga_dsdl/pretrained_dga_detection.ipynb b/notebooks/pretrained_dga_model_dsdl.ipynb similarity index 95% rename from notebooks/dga_dsdl/pretrained_dga_detection.ipynb rename to notebooks/pretrained_dga_model_dsdl.ipynb index 161b682b3a..4db03ba210 100644 --- a/notebooks/dga_dsdl/pretrained_dga_detection.ipynb +++ b/notebooks/pretrained_dga_model_dsdl.ipynb @@ -119,7 +119,7 @@ "outputs": [], "source": [ "# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes\n", - "df, param = stage(\"dga_model_dltk\")\n", + "df, param = stage(\"pretrained_dga_model_dsdl\")\n", "print(df.describe())\n", "print(param)" ] @@ -144,7 +144,7 @@ "# available inputs: data and parameters\n", "# returns the model object which will be used as a reference to call fit, apply and summary subsequently\n", "def init(df,param):\n", - " model = tf.keras.models.load_model(MODEL_DIRECTORY + \"pretrained_dga_detection_dga_model_dltk\")\n", + " model = tf.keras.models.load_model(MODEL_DIRECTORY + \"pretrained_dga_model_dsdl\")\n", " return model" ] }, @@ -212,13 +212,13 @@ "source": [ "# apply your model\n", "# returns the calculated results\n", - "tokenizer = pickle.load(open(MODEL_DIRECTORY+\"pretrained_dga_detection_dga_model_dltk/tokenizer\", 'rb'))\n", - "vectorizer_words = pickle.load(open(MODEL_DIRECTORY+\"pretrained_dga_detection_dga_model_dltk/vectorizer_words\", 'rb'))\n", - "vectorizer_non_dga_domains = pickle.load(open(MODEL_DIRECTORY+\"pretrained_dga_detection_dga_model_dltk/vectorizer_non_dga_domains\", 'rb'))\n", - "weight_words = pickle.load(open(MODEL_DIRECTORY+\"pretrained_dga_detection_dga_model_dltk/weight_words\", 'rb'))\n", - "weight_non_dga_grams = pickle.load(open(MODEL_DIRECTORY+\"pretrained_dga_detection_dga_model_dltk/weight_non_dga_grams\", 'rb'))\n", - "domains = pickle.load(open(MODEL_DIRECTORY+\"pretrained_dga_detection_dga_model_dltk/domains\", 'rb'))\n", - "model = tf.keras.models.load_model(MODEL_DIRECTORY + \"pretrained_dga_detection_dga_model_dltk\")\n", + "tokenizer = pickle.load(open(MODEL_DIRECTORY+\"pretrained_dga_model_dsdl/tokenizer\", 'rb'))\n", + "vectorizer_words = pickle.load(open(MODEL_DIRECTORY+\"pretrained_dga_model_dsdl/vectorizer_words\", 'rb'))\n", + "vectorizer_non_dga_domains = pickle.load(open(MODEL_DIRECTORY+\"pretrained_dga_model_dsdl/vectorizer_non_dga_domains\", 'rb'))\n", + "weight_words = pickle.load(open(MODEL_DIRECTORY+\"pretrained_dga_model_dsdl/weight_words\", 'rb'))\n", + "weight_non_dga_grams = pickle.load(open(MODEL_DIRECTORY+\"pretrained_dga_model_dsdl/weight_non_dga_grams\", 'rb'))\n", + "domains = pickle.load(open(MODEL_DIRECTORY+\"pretrained_dga_model_dsdl/domains\", 'rb'))\n", + "model = tf.keras.models.load_model(MODEL_DIRECTORY + \"pretrained_dga_model_dsdl\")\n", "\n", "def entropy(domain):\n", " p, lns = Counter(domain), float(len(domain))\n", @@ -318,7 +318,7 @@ "source": [ "# load model from name in expected convention \"_\"\n", "def load(name):\n", - " model = tf.keras.models.load_model(MODEL_DIRECTORY + \"pretrained_dga_detection_dga_model_dltk\")\n", + " model = tf.keras.models.load_model(MODEL_DIRECTORY + \"pretrained_dga_model_dsdl\")\n", " return model" ] }, diff --git a/notebooks/pretrained_dga_model_dsdl.json b/notebooks/pretrained_dga_model_dsdl.json new file mode 100644 index 0000000000..5b026f0a7d --- /dev/null +++ b/notebooks/pretrained_dga_model_dsdl.json @@ -0,0 +1 @@ +{"options": {"params": {"mode": "stage", "algo": "pretrained_dga_model_dsdl"}, "args": ["is_dga", "domain"], "target_variable": ["is_dga"], "feature_variables": ["domain"], "model_name": "pretrained_dga_model_dsdl", "algo_name": "MLTKContainer", "mlspl_limits": {"handle_new_cat": "default", "max_distinct_cat_values": "100", "max_distinct_cat_values_for_classifiers": "100", "max_distinct_cat_values_for_scoring": "100", "max_fit_time": "600", "max_inputs": "100000", "max_memory_usage_mb": "4000", "max_model_size_mb": "30", "max_score_time": "600", "use_sampling": "true"}, "kfold_cv": null}, "feature_variables": ["domain"], "target_variables": ["is_dga"]} \ No newline at end of file From 902221712be07b10b0c3ec07eb6270cd5acaead3 Mon Sep 17 00:00:00 2001 From: Namratha Sreekanta Date: Mon, 26 Sep 2022 12:26:31 -0700 Subject: [PATCH 14/24] making changes to how to implement --- ...ect_dga_domains_using_pretrained_model_in_dsdl.yml | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml b/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml index 576e55287a..3052cc01d8 100644 --- a/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml +++ b/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml @@ -9,7 +9,16 @@ datamodel: description: 'The following analytic uses a pretrained deep learning model to detect Domain Generation Algorithm (DGA) generated domains. The model is trained independently and then made available in AWS S3. One of the prominent indicators of a domain being DGA generated is if the domain name consists of unusual character sequences or concatenated dictionary words. Adversaries often use clever techniques to obfuscate machine generated domain names as human generated. Predicting DGA generated domain names requires analysis and building a model based on carefully chosen features. The deep learning model we have developed uses the domain name to anlayze patterns of character sequences along with carefully chosen custom features to predict if a domain is DGA generated. The model takes a domain name consisting of second-level and top-level domain names as input and outputs a dga_score. Higher the dga_score, the more likely the input domain is a DGA domain. The threshold for flagging a domain as DGA is set at 0.5.' search: '| tstats `security_content_summariesonly` values(DNS.answer) as IPs min(_time) as firstTime max(_time) as lastTime from datamodel=Network_Resolution by DNS.src, DNS.query | `drop_dm_object_name("DNS")` | rename query AS domain | fields IPs, src, domain, firstTime, lastTime, src | apply pretrained_dga_model_dsdl | rename pred_dga_proba AS dga_score | where dga_score>0.5 | `security_content_ctime(firstTime)` | `security_content_ctime(lastTime)` | table src, domain, IPs, firstTime, lastTime, dga_score | `detect_dga_domains_using_pretrained_model_in_dsdl_filter`' -how_to_implement: This detection depends on the DSDL app which can be found here - https://splunkbase.splunk.com/app/4607/ and the Network Resolution datamodel which can be found here - https://splunkbase.splunk.com/app/1621/. The dga_score threshold is set at 0.5 and this should be adjusted based on the environment. The detection uses a pre-trained deep learning model that needs to be deployed in DSDL. Steps for deployment are outlined here - https://github.com/splunk/security_content/notebooks/dga_dsdl/README.md. +how_to_implement: " +* Steps to deploy DGA detection model into Splunk App DSDL. This detection depends on the Splunk app for Data Science and Deep Learning which can be found here - https://splunkbase.splunk.com/app/4607/ and the Network Resolution datamodel which can be found here - https://splunkbase.splunk.com/app/1621/. The detection uses a pre-trained deep learning model that needs to be deployed in DSDL app. Follow the steps for deployment here - https://github.com/splunk/security_content/wiki/How-to-deploy-Deep-Learning-models-for-ESCU. +* Download the artifacts .tar.gz file from S3 bucket https://splunk-seal.s3.us-west-2.amazonaws.com/pretrained_dga_model_dsdl.tar.gz +* Download pretrained__model_dsdl.ipynb from https://github.com/splunk/security_content/notebooks +* Below steps need to be followed inside Jupyter lab + * Upload the pretrained_dga_model_dsdl.tar.gz file into app/model/data path using the upload option in the jupyter notebook. + * Untar the artifact pretrained_dga_model_dsdl.tar.gz using tar -xf app/model/data/pretrained_dga_model_dsdl.tar.gz -C app/model/data + * Upload pretrained_dga_model_dsdl.pynb into Jupyter lab notebooks folder using the upload option in Jupyter lab + * Save the notebook using the save option in jupyter notebook. + * Upload pretrained_dga_model_dsdl.json into notebooks/data folder." known_false_positives: False positives may be present if domain name is similar to dga generated domains. references: - https://attack.mitre.org/techniques/T1568/002/ From 2ba2bc8e5396dbe90e3322a6b3190e4cd5e7489a Mon Sep 17 00:00:00 2001 From: Bhavin Patel Date: Mon, 26 Sep 2022 16:41:32 -0500 Subject: [PATCH 15/24] Update detect_dga_domains_using_pretrained_model_in_dsdl.yml updating story names --- .../detect_dga_domains_using_pretrained_model_in_dsdl.yml | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml b/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml index 3052cc01d8..f620692684 100644 --- a/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml +++ b/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml @@ -26,8 +26,7 @@ references: - https://en.wikipedia.org/wiki/Domain_generation_algorithm tags: analytic_story: - - Data Protection - - Prohibited Traffic Allowed or Protocol Mismatch + - Data Exfiltration - DNS Hijacking - Suspicious DNS Traffic - Dynamic DNS @@ -72,4 +71,4 @@ tags: - firstTime - lastTime risk_score: 63 - security_domain: Network \ No newline at end of file + security_domain: Network From 179e58e0870c79668edb0f19f0d3f70a3195c599 Mon Sep 17 00:00:00 2001 From: Namratha Sreekanta Date: Wed, 28 Sep 2022 12:36:40 -0700 Subject: [PATCH 16/24] changing files --- lookups/__mlspl_dga_model_dltk.mlmodel | 2 -- lookups/__mlspl_pretrained_dga_model_dsdl.mlmodel | 2 ++ ...dga_model_dltk.yml => __mlspl_pretrained_dga_model_dsdl.yml} | 0 3 files changed, 2 insertions(+), 2 deletions(-) delete mode 100644 lookups/__mlspl_dga_model_dltk.mlmodel create mode 100644 lookups/__mlspl_pretrained_dga_model_dsdl.mlmodel rename lookups/{__mlspl_dga_model_dltk.yml => __mlspl_pretrained_dga_model_dsdl.yml} (100%) diff --git a/lookups/__mlspl_dga_model_dltk.mlmodel b/lookups/__mlspl_dga_model_dltk.mlmodel deleted file mode 100644 index 9f9c65d6b9..0000000000 --- a/lookups/__mlspl_dga_model_dltk.mlmodel +++ /dev/null @@ -1,2 +0,0 @@ -algo,model,options -MLTKContainer,"{""__mlspl_type"": [""mltkc.MLTKContainer"", ""MLTKContainer""], ""dict"": {""endpoint_url"": ""https://localhost:58784"", ""out_params"": {""params"": {""mode"": ""stage"", ""algo"": ""pretrained_dga_detection""}, ""args"": [""is_dga"", ""domain""], ""target_variable"": [""is_dga""], ""feature_variables"": [""domain""], ""model_name"": ""dga_model_dltk"", ""algo_name"": ""MLTKContainer"", ""mlspl_limits"": {""handle_new_cat"": ""default"", ""max_distinct_cat_values"": ""100"", ""max_distinct_cat_values_for_classifiers"": ""100"", ""max_distinct_cat_values_for_scoring"": ""100"", ""max_fit_time"": ""6000"", ""max_inputs"": ""1000000"", ""max_memory_usage_mb"": ""4000"", ""max_model_size_mb"": ""30"", ""max_score_time"": ""6000"", ""use_sampling"": ""true""}, ""kfold_cv"": null}, ""feature_variables"": [""domain""], ""target_variable"": ""is_dga""}}","{""params"": {""mode"": ""stage"", ""algo"": ""pretrained_dga_detection""}, ""args"": [""is_dga"", ""domain""], ""target_variable"": [""is_dga""], ""feature_variables"": [""domain""], ""model_name"": ""dga_model_dltk"", ""algo_name"": ""MLTKContainer"", ""mlspl_limits"": {""handle_new_cat"": ""default"", ""max_distinct_cat_values"": ""100"", ""max_distinct_cat_values_for_classifiers"": ""100"", ""max_distinct_cat_values_for_scoring"": ""100"", ""max_fit_time"": ""6000"", ""max_inputs"": ""1000000"", ""max_memory_usage_mb"": ""4000"", ""max_model_size_mb"": ""30"", ""max_score_time"": ""6000"", ""use_sampling"": ""true""}, ""kfold_cv"": null}" diff --git a/lookups/__mlspl_pretrained_dga_model_dsdl.mlmodel b/lookups/__mlspl_pretrained_dga_model_dsdl.mlmodel new file mode 100644 index 0000000000..3e27dc8bd8 --- /dev/null +++ b/lookups/__mlspl_pretrained_dga_model_dsdl.mlmodel @@ -0,0 +1,2 @@ +algo,model,options +MLTKContainer,"{""__mlspl_type"": [""mltkc.MLTKContainer"", ""MLTKContainer""], ""dict"": {""endpoint_url"": ""https://localhost:53378"", ""out_params"": {""params"": {""mode"": ""stage"", ""algo"": ""pretrained_dga_model_dsdl""}, ""args"": [""is_dga"", ""domain""], ""target_variable"": [""is_dga""], ""feature_variables"": [""domain""], ""model_name"": ""pretrained_dga_model_dsdl"", ""algo_name"": ""MLTKContainer"", ""mlspl_limits"": {""handle_new_cat"": ""default"", ""max_distinct_cat_values"": ""100"", ""max_distinct_cat_values_for_classifiers"": ""100"", ""max_distinct_cat_values_for_scoring"": ""100"", ""max_fit_time"": ""600"", ""max_inputs"": ""100000"", ""max_memory_usage_mb"": ""4000"", ""max_model_size_mb"": ""30"", ""max_score_time"": ""600"", ""use_sampling"": ""true""}, ""kfold_cv"": null}, ""feature_variables"": [""domain""], ""target_variable"": ""is_dga""}}","{""params"": {""mode"": ""stage"", ""algo"": ""pretrained_dga_model_dsdl""}, ""args"": [""is_dga"", ""domain""], ""target_variable"": [""is_dga""], ""feature_variables"": [""domain""], ""model_name"": ""pretrained_dga_model_dsdl"", ""algo_name"": ""MLTKContainer"", ""mlspl_limits"": {""handle_new_cat"": ""default"", ""max_distinct_cat_values"": ""100"", ""max_distinct_cat_values_for_classifiers"": ""100"", ""max_distinct_cat_values_for_scoring"": ""100"", ""max_fit_time"": ""600"", ""max_inputs"": ""100000"", ""max_memory_usage_mb"": ""4000"", ""max_model_size_mb"": ""30"", ""max_score_time"": ""600"", ""use_sampling"": ""true""}, ""kfold_cv"": null}" diff --git a/lookups/__mlspl_dga_model_dltk.yml b/lookups/__mlspl_pretrained_dga_model_dsdl.yml similarity index 100% rename from lookups/__mlspl_dga_model_dltk.yml rename to lookups/__mlspl_pretrained_dga_model_dsdl.yml From 84ae1743fe41767786680d351e84d20832757e16 Mon Sep 17 00:00:00 2001 From: Namratha Sreekanta Date: Wed, 28 Sep 2022 13:16:22 -0700 Subject: [PATCH 17/24] changing file names --- lookups/__mlspl_pretrained_dga_model_dsdl.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/lookups/__mlspl_pretrained_dga_model_dsdl.yml b/lookups/__mlspl_pretrained_dga_model_dsdl.yml index ee32c8a0b7..9aba0dcf66 100644 --- a/lookups/__mlspl_pretrained_dga_model_dsdl.yml +++ b/lookups/__mlspl_pretrained_dga_model_dsdl.yml @@ -1,4 +1,4 @@ -description: Detect DGA domains using Pretrained Model in DLTK -filename: __mlspl_dga_model_dltk.mlmodel -name: __mlspl_dga_model_dltk +description: Detect DGA domains using Pretrained Model in DSDL +filename: __mlspl_pretrained_dga_model_dsdl.mlmodel +name: __mlspl_pretrained_dga_model_dsdl case_sensitive_match: 'false' \ No newline at end of file From 70753ea779f1b5dc6ef38bbea1bbac8dc8e71506 Mon Sep 17 00:00:00 2001 From: Namratha Sreekanta Date: Wed, 28 Sep 2022 15:24:43 -0700 Subject: [PATCH 18/24] minor desc change in .yml file --- .../detect_dga_domains_using_pretrained_model_in_dsdl.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml b/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml index f620692684..bff50ecdf0 100644 --- a/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml +++ b/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml @@ -12,7 +12,7 @@ search: '| tstats `security_content_summariesonly` values(DNS.answer) as IPs min how_to_implement: " * Steps to deploy DGA detection model into Splunk App DSDL. This detection depends on the Splunk app for Data Science and Deep Learning which can be found here - https://splunkbase.splunk.com/app/4607/ and the Network Resolution datamodel which can be found here - https://splunkbase.splunk.com/app/1621/. The detection uses a pre-trained deep learning model that needs to be deployed in DSDL app. Follow the steps for deployment here - https://github.com/splunk/security_content/wiki/How-to-deploy-Deep-Learning-models-for-ESCU. * Download the artifacts .tar.gz file from S3 bucket https://splunk-seal.s3.us-west-2.amazonaws.com/pretrained_dga_model_dsdl.tar.gz -* Download pretrained__model_dsdl.ipynb from https://github.com/splunk/security_content/notebooks +* Download pretrained_dga_model_dsdl.ipynb from https://github.com/splunk/security_content/notebooks * Below steps need to be followed inside Jupyter lab * Upload the pretrained_dga_model_dsdl.tar.gz file into app/model/data path using the upload option in the jupyter notebook. * Untar the artifact pretrained_dga_model_dsdl.tar.gz using tar -xf app/model/data/pretrained_dga_model_dsdl.tar.gz -C app/model/data From 23418cd595555518315e784215c33abed2c77ec4 Mon Sep 17 00:00:00 2001 From: Namratha Sreekanta Date: Wed, 28 Sep 2022 18:08:40 -0700 Subject: [PATCH 19/24] Changing the links for S3 bucket --- .../detect_dga_domains_using_pretrained_model_in_dsdl.yml | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml b/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml index bff50ecdf0..378000e7a7 100644 --- a/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml +++ b/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml @@ -11,8 +11,9 @@ search: '| tstats `security_content_summariesonly` values(DNS.answer) as IPs min | `drop_dm_object_name("DNS")` | rename query AS domain | fields IPs, src, domain, firstTime, lastTime, src | apply pretrained_dga_model_dsdl | rename pred_dga_proba AS dga_score | where dga_score>0.5 | `security_content_ctime(firstTime)` | `security_content_ctime(lastTime)` | table src, domain, IPs, firstTime, lastTime, dga_score | `detect_dga_domains_using_pretrained_model_in_dsdl_filter`' how_to_implement: " * Steps to deploy DGA detection model into Splunk App DSDL. This detection depends on the Splunk app for Data Science and Deep Learning which can be found here - https://splunkbase.splunk.com/app/4607/ and the Network Resolution datamodel which can be found here - https://splunkbase.splunk.com/app/1621/. The detection uses a pre-trained deep learning model that needs to be deployed in DSDL app. Follow the steps for deployment here - https://github.com/splunk/security_content/wiki/How-to-deploy-Deep-Learning-models-for-ESCU. -* Download the artifacts .tar.gz file from S3 bucket https://splunk-seal.s3.us-west-2.amazonaws.com/pretrained_dga_model_dsdl.tar.gz -* Download pretrained_dga_model_dsdl.ipynb from https://github.com/splunk/security_content/notebooks +* Download the artifacts .tar.gz file from the S3 bucket - http://seal.splunkresearch.com/pretrained_dga_model_dsdl.tar.gz +* Download the pretrained_dga_model_dsdl.ipynb Jupyter notebook from https://github.com/splunk/security_content/notebooks +* Login to the Jupyter Lab for pretrained_dga_model_dsdl container. This container should be listed on Containers page for DSDL app. * Below steps need to be followed inside Jupyter lab * Upload the pretrained_dga_model_dsdl.tar.gz file into app/model/data path using the upload option in the jupyter notebook. * Untar the artifact pretrained_dga_model_dsdl.tar.gz using tar -xf app/model/data/pretrained_dga_model_dsdl.tar.gz -C app/model/data From 6cfc24babc46debfd2586c1dce111c2ef6eb7278 Mon Sep 17 00:00:00 2001 From: Namratha Sreekanta Date: Thu, 29 Sep 2022 10:59:09 -0700 Subject: [PATCH 20/24] changing the bucket name --- .../detect_dga_domains_using_pretrained_model_in_dsdl.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml b/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml index 378000e7a7..887a377ea5 100644 --- a/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml +++ b/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml @@ -11,7 +11,7 @@ search: '| tstats `security_content_summariesonly` values(DNS.answer) as IPs min | `drop_dm_object_name("DNS")` | rename query AS domain | fields IPs, src, domain, firstTime, lastTime, src | apply pretrained_dga_model_dsdl | rename pred_dga_proba AS dga_score | where dga_score>0.5 | `security_content_ctime(firstTime)` | `security_content_ctime(lastTime)` | table src, domain, IPs, firstTime, lastTime, dga_score | `detect_dga_domains_using_pretrained_model_in_dsdl_filter`' how_to_implement: " * Steps to deploy DGA detection model into Splunk App DSDL. This detection depends on the Splunk app for Data Science and Deep Learning which can be found here - https://splunkbase.splunk.com/app/4607/ and the Network Resolution datamodel which can be found here - https://splunkbase.splunk.com/app/1621/. The detection uses a pre-trained deep learning model that needs to be deployed in DSDL app. Follow the steps for deployment here - https://github.com/splunk/security_content/wiki/How-to-deploy-Deep-Learning-models-for-ESCU. -* Download the artifacts .tar.gz file from the S3 bucket - http://seal.splunkresearch.com/pretrained_dga_model_dsdl.tar.gz +* Download the artifacts .tar.gz file from the S3 bucket - https://seal.splunkresearch.com.s3-us-west-2.amazonaws.com/pretrained_dga_model_dsdl.tar.gz * Download the pretrained_dga_model_dsdl.ipynb Jupyter notebook from https://github.com/splunk/security_content/notebooks * Login to the Jupyter Lab for pretrained_dga_model_dsdl container. This container should be listed on Containers page for DSDL app. * Below steps need to be followed inside Jupyter lab From 9c22497ba5309430194045de692eaa4390dcbb73 Mon Sep 17 00:00:00 2001 From: Namratha Sreekanta Date: Thu, 29 Sep 2022 11:36:31 -0700 Subject: [PATCH 21/24] Changing the link --- .../detect_dga_domains_using_pretrained_model_in_dsdl.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml b/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml index 887a377ea5..77eb45f74c 100644 --- a/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml +++ b/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml @@ -6,12 +6,12 @@ author: Abhinav Mishra, Kumar Sharad and Namratha Sreekanta, Splunk type: Anomaly datamodel: - Network_Resolution -description: 'The following analytic uses a pretrained deep learning model to detect Domain Generation Algorithm (DGA) generated domains. The model is trained independently and then made available in AWS S3. One of the prominent indicators of a domain being DGA generated is if the domain name consists of unusual character sequences or concatenated dictionary words. Adversaries often use clever techniques to obfuscate machine generated domain names as human generated. Predicting DGA generated domain names requires analysis and building a model based on carefully chosen features. The deep learning model we have developed uses the domain name to anlayze patterns of character sequences along with carefully chosen custom features to predict if a domain is DGA generated. The model takes a domain name consisting of second-level and top-level domain names as input and outputs a dga_score. Higher the dga_score, the more likely the input domain is a DGA domain. The threshold for flagging a domain as DGA is set at 0.5.' +description: 'The following analytic uses a pretrained deep learning model to detect Domain Generation Algorithm (DGA) generated domains. The model is trained independently and is then made available for download. One of the prominent indicators of a domain being DGA generated is if the domain name consists of unusual character sequences or concatenated dictionary words. Adversaries often use clever techniques to obfuscate machine generated domain names as human generated. Predicting DGA generated domain names requires analysis and building a model based on carefully chosen features. The deep learning model we have developed uses the domain name to anlayze patterns of character sequences along with carefully chosen custom features to predict if a domain is DGA generated. The model takes a domain name consisting of second-level and top-level domain names as input and outputs a dga_score. Higher the dga_score, the more likely the input domain is a DGA domain. The threshold for flagging a domain as DGA is set at 0.5.' search: '| tstats `security_content_summariesonly` values(DNS.answer) as IPs min(_time) as firstTime max(_time) as lastTime from datamodel=Network_Resolution by DNS.src, DNS.query | `drop_dm_object_name("DNS")` | rename query AS domain | fields IPs, src, domain, firstTime, lastTime, src | apply pretrained_dga_model_dsdl | rename pred_dga_proba AS dga_score | where dga_score>0.5 | `security_content_ctime(firstTime)` | `security_content_ctime(lastTime)` | table src, domain, IPs, firstTime, lastTime, dga_score | `detect_dga_domains_using_pretrained_model_in_dsdl_filter`' how_to_implement: " * Steps to deploy DGA detection model into Splunk App DSDL. This detection depends on the Splunk app for Data Science and Deep Learning which can be found here - https://splunkbase.splunk.com/app/4607/ and the Network Resolution datamodel which can be found here - https://splunkbase.splunk.com/app/1621/. The detection uses a pre-trained deep learning model that needs to be deployed in DSDL app. Follow the steps for deployment here - https://github.com/splunk/security_content/wiki/How-to-deploy-Deep-Learning-models-for-ESCU. -* Download the artifacts .tar.gz file from the S3 bucket - https://seal.splunkresearch.com.s3-us-west-2.amazonaws.com/pretrained_dga_model_dsdl.tar.gz +* Download the artifacts .tar.gz file from the link - https://seal.splunkresearch.com/pretrained_dga_model_dsdl.tar.gz * Download the pretrained_dga_model_dsdl.ipynb Jupyter notebook from https://github.com/splunk/security_content/notebooks * Login to the Jupyter Lab for pretrained_dga_model_dsdl container. This container should be listed on Containers page for DSDL app. * Below steps need to be followed inside Jupyter lab From 8c527b79e3321b69cdf1a4ebc30dd1d99d57ece8 Mon Sep 17 00:00:00 2001 From: Bhavin Patel Date: Thu, 29 Sep 2022 13:06:04 -0700 Subject: [PATCH 22/24] Update detect_dga_domains_using_pretrained_model_in_dsdl.yml --- .../detect_dga_domains_using_pretrained_model_in_dsdl.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml b/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml index 77eb45f74c..96a156c0b1 100644 --- a/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml +++ b/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml @@ -44,7 +44,7 @@ tags: impact: 70 kill_chain_phases: - Command & Control - message: A potentially a DGA domain $domain$ was detected, kindly review. + message: A potentially a DGA domain $domain$ was detected from host $src$, kindly review. mitre_attack_id: - T1568.002 nist: From de34ea116fac5834bdf32fed450d28a9a3c9dcf2 Mon Sep 17 00:00:00 2001 From: Bhavin Patel Date: Thu, 29 Sep 2022 13:09:04 -0700 Subject: [PATCH 23/24] Update detect_dga_domains_using_pretrained_model_in_dsdl.yml --- .../detect_dga_domains_using_pretrained_model_in_dsdl.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml b/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml index 96a156c0b1..440f3d94fc 100644 --- a/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml +++ b/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml @@ -32,7 +32,7 @@ tags: - Suspicious DNS Traffic - Dynamic DNS - Command and Control - asset_type: Web Server + asset_type: Endpoint cis20: - CIS 8 - CIS 12 @@ -44,7 +44,7 @@ tags: impact: 70 kill_chain_phases: - Command & Control - message: A potentially a DGA domain $domain$ was detected from host $src$, kindly review. + message: A potential connection to a DGA domain $domain$ was detected from host $src$, kindly review. mitre_attack_id: - T1568.002 nist: @@ -54,7 +54,7 @@ tags: - DE.CM observable: - name: domain - type: Other + type: URL Domain role: - Attacker - name: src From 8148a767ca31328a93e9b24880c84b2ba869e001 Mon Sep 17 00:00:00 2001 From: Bhavin Patel Date: Thu, 29 Sep 2022 13:53:00 -0700 Subject: [PATCH 24/24] Update detect_dga_domains_using_pretrained_model_in_dsdl.yml minor updates --- .../detect_dga_domains_using_pretrained_model_in_dsdl.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml b/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml index 440f3d94fc..229779b2e6 100644 --- a/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml +++ b/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml @@ -8,7 +8,7 @@ datamodel: - Network_Resolution description: 'The following analytic uses a pretrained deep learning model to detect Domain Generation Algorithm (DGA) generated domains. The model is trained independently and is then made available for download. One of the prominent indicators of a domain being DGA generated is if the domain name consists of unusual character sequences or concatenated dictionary words. Adversaries often use clever techniques to obfuscate machine generated domain names as human generated. Predicting DGA generated domain names requires analysis and building a model based on carefully chosen features. The deep learning model we have developed uses the domain name to anlayze patterns of character sequences along with carefully chosen custom features to predict if a domain is DGA generated. The model takes a domain name consisting of second-level and top-level domain names as input and outputs a dga_score. Higher the dga_score, the more likely the input domain is a DGA domain. The threshold for flagging a domain as DGA is set at 0.5.' search: '| tstats `security_content_summariesonly` values(DNS.answer) as IPs min(_time) as firstTime max(_time) as lastTime from datamodel=Network_Resolution by DNS.src, DNS.query -| `drop_dm_object_name("DNS")` | rename query AS domain | fields IPs, src, domain, firstTime, lastTime, src | apply pretrained_dga_model_dsdl | rename pred_dga_proba AS dga_score | where dga_score>0.5 | `security_content_ctime(firstTime)` | `security_content_ctime(lastTime)` | table src, domain, IPs, firstTime, lastTime, dga_score | `detect_dga_domains_using_pretrained_model_in_dsdl_filter`' +| `drop_dm_object_name("DNS")` | rename query AS domain | fields IPs, src, domain, firstTime, lastTime | apply pretrained_dga_model_dsdl | rename pred_dga_proba AS dga_score | where dga_score>0.5 | `security_content_ctime(firstTime)` | `security_content_ctime(lastTime)` | table src, domain, IPs, firstTime, lastTime, dga_score | `detect_dga_domains_using_pretrained_model_in_dsdl_filter`' how_to_implement: " * Steps to deploy DGA detection model into Splunk App DSDL. This detection depends on the Splunk app for Data Science and Deep Learning which can be found here - https://splunkbase.splunk.com/app/4607/ and the Network Resolution datamodel which can be found here - https://splunkbase.splunk.com/app/1621/. The detection uses a pre-trained deep learning model that needs to be deployed in DSDL app. Follow the steps for deployment here - https://github.com/splunk/security_content/wiki/How-to-deploy-Deep-Learning-models-for-ESCU. * Download the artifacts .tar.gz file from the link - https://seal.splunkresearch.com/pretrained_dga_model_dsdl.tar.gz