diff --git a/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml b/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml new file mode 100644 index 0000000000..229779b2e6 --- /dev/null +++ b/detections/experimental/network/detect_dga_domains_using_pretrained_model_in_dsdl.yml @@ -0,0 +1,75 @@ +name: Detect DGA domains using pretrained model in DSDL +id: 92e24f32-9b9a-4060-bba2-2a0eb3bf3493 +version: 1 +date: '2022-09-15' +author: Abhinav Mishra, Kumar Sharad and Namratha Sreekanta, Splunk +type: Anomaly +datamodel: +- Network_Resolution +description: 'The following analytic uses a pretrained deep learning model to detect Domain Generation Algorithm (DGA) generated domains. The model is trained independently and is then made available for download. One of the prominent indicators of a domain being DGA generated is if the domain name consists of unusual character sequences or concatenated dictionary words. Adversaries often use clever techniques to obfuscate machine generated domain names as human generated. Predicting DGA generated domain names requires analysis and building a model based on carefully chosen features. The deep learning model we have developed uses the domain name to anlayze patterns of character sequences along with carefully chosen custom features to predict if a domain is DGA generated. The model takes a domain name consisting of second-level and top-level domain names as input and outputs a dga_score. Higher the dga_score, the more likely the input domain is a DGA domain. The threshold for flagging a domain as DGA is set at 0.5.' +search: '| tstats `security_content_summariesonly` values(DNS.answer) as IPs min(_time) as firstTime max(_time) as lastTime from datamodel=Network_Resolution by DNS.src, DNS.query +| `drop_dm_object_name("DNS")` | rename query AS domain | fields IPs, src, domain, firstTime, lastTime | apply pretrained_dga_model_dsdl | rename pred_dga_proba AS dga_score | where dga_score>0.5 | `security_content_ctime(firstTime)` | `security_content_ctime(lastTime)` | table src, domain, IPs, firstTime, lastTime, dga_score | `detect_dga_domains_using_pretrained_model_in_dsdl_filter`' +how_to_implement: " +* Steps to deploy DGA detection model into Splunk App DSDL. This detection depends on the Splunk app for Data Science and Deep Learning which can be found here - https://splunkbase.splunk.com/app/4607/ and the Network Resolution datamodel which can be found here - https://splunkbase.splunk.com/app/1621/. The detection uses a pre-trained deep learning model that needs to be deployed in DSDL app. Follow the steps for deployment here - https://github.com/splunk/security_content/wiki/How-to-deploy-Deep-Learning-models-for-ESCU. +* Download the artifacts .tar.gz file from the link - https://seal.splunkresearch.com/pretrained_dga_model_dsdl.tar.gz +* Download the pretrained_dga_model_dsdl.ipynb Jupyter notebook from https://github.com/splunk/security_content/notebooks +* Login to the Jupyter Lab for pretrained_dga_model_dsdl container. This container should be listed on Containers page for DSDL app. +* Below steps need to be followed inside Jupyter lab + * Upload the pretrained_dga_model_dsdl.tar.gz file into app/model/data path using the upload option in the jupyter notebook. + * Untar the artifact pretrained_dga_model_dsdl.tar.gz using tar -xf app/model/data/pretrained_dga_model_dsdl.tar.gz -C app/model/data + * Upload pretrained_dga_model_dsdl.pynb into Jupyter lab notebooks folder using the upload option in Jupyter lab + * Save the notebook using the save option in jupyter notebook. + * Upload pretrained_dga_model_dsdl.json into notebooks/data folder." +known_false_positives: False positives may be present if domain name is similar to dga generated domains. +references: +- https://attack.mitre.org/techniques/T1568/002/ +- https://unit42.paloaltonetworks.com/threat-brief-understanding-domain-generation-algorithms-dga/ +- https://en.wikipedia.org/wiki/Domain_generation_algorithm +tags: + analytic_story: + - Data Exfiltration + - DNS Hijacking + - Suspicious DNS Traffic + - Dynamic DNS + - Command and Control + asset_type: Endpoint + cis20: + - CIS 8 + - CIS 12 + - CIS 13 + confidence: 90 + context: + - Source:Endpoint + - Stage:Command And Control + impact: 70 + kill_chain_phases: + - Command & Control + message: A potential connection to a DGA domain $domain$ was detected from host $src$, kindly review. + mitre_attack_id: + - T1568.002 + nist: + - PR.DS + - PR.PT + - DE.AE + - DE.CM + observable: + - name: domain + type: URL Domain + role: + - Attacker + - name: src + type: Hostname + role: + - Victim + product: + - Splunk Enterprise + - Splunk Enterprise Security + - Splunk Cloud + required_fields: + - IPs + - src + - domain + - firstTime + - lastTime + risk_score: 63 + security_domain: Network diff --git a/lookups/__mlspl_pretrained_dga_model_dsdl.mlmodel b/lookups/__mlspl_pretrained_dga_model_dsdl.mlmodel new file mode 100644 index 0000000000..3e27dc8bd8 --- /dev/null +++ b/lookups/__mlspl_pretrained_dga_model_dsdl.mlmodel @@ -0,0 +1,2 @@ +algo,model,options +MLTKContainer,"{""__mlspl_type"": [""mltkc.MLTKContainer"", ""MLTKContainer""], ""dict"": {""endpoint_url"": ""https://localhost:53378"", ""out_params"": {""params"": {""mode"": ""stage"", ""algo"": ""pretrained_dga_model_dsdl""}, ""args"": [""is_dga"", ""domain""], ""target_variable"": [""is_dga""], ""feature_variables"": [""domain""], ""model_name"": ""pretrained_dga_model_dsdl"", ""algo_name"": ""MLTKContainer"", ""mlspl_limits"": {""handle_new_cat"": ""default"", ""max_distinct_cat_values"": ""100"", ""max_distinct_cat_values_for_classifiers"": ""100"", ""max_distinct_cat_values_for_scoring"": ""100"", ""max_fit_time"": ""600"", ""max_inputs"": ""100000"", ""max_memory_usage_mb"": ""4000"", ""max_model_size_mb"": ""30"", ""max_score_time"": ""600"", ""use_sampling"": ""true""}, ""kfold_cv"": null}, ""feature_variables"": [""domain""], ""target_variable"": ""is_dga""}}","{""params"": {""mode"": ""stage"", ""algo"": ""pretrained_dga_model_dsdl""}, ""args"": [""is_dga"", ""domain""], ""target_variable"": [""is_dga""], ""feature_variables"": [""domain""], ""model_name"": ""pretrained_dga_model_dsdl"", ""algo_name"": ""MLTKContainer"", ""mlspl_limits"": {""handle_new_cat"": ""default"", ""max_distinct_cat_values"": ""100"", ""max_distinct_cat_values_for_classifiers"": ""100"", ""max_distinct_cat_values_for_scoring"": ""100"", ""max_fit_time"": ""600"", ""max_inputs"": ""100000"", ""max_memory_usage_mb"": ""4000"", ""max_model_size_mb"": ""30"", ""max_score_time"": ""600"", ""use_sampling"": ""true""}, ""kfold_cv"": null}" diff --git a/lookups/__mlspl_pretrained_dga_model_dsdl.yml b/lookups/__mlspl_pretrained_dga_model_dsdl.yml new file mode 100644 index 0000000000..9aba0dcf66 --- /dev/null +++ b/lookups/__mlspl_pretrained_dga_model_dsdl.yml @@ -0,0 +1,4 @@ +description: Detect DGA domains using Pretrained Model in DSDL +filename: __mlspl_pretrained_dga_model_dsdl.mlmodel +name: __mlspl_pretrained_dga_model_dsdl +case_sensitive_match: 'false' \ No newline at end of file diff --git a/notebooks/pretrained_dga_detection.yml b/notebooks/pretrained_dga_detection.yml new file mode 100644 index 0000000000..79979162bf --- /dev/null +++ b/notebooks/pretrained_dga_detection.yml @@ -0,0 +1,3 @@ +description: A Jupyter notebook for detecting DGA domains using a pre-trained model in DSDL +filename: pretrained_dga_model_dsdl.ipynb +name: pretrained_dga_model_dsdl \ No newline at end of file diff --git a/notebooks/pretrained_dga_model_dsdl.ipynb b/notebooks/pretrained_dga_model_dsdl.ipynb new file mode 100644 index 0000000000..4db03ba210 --- /dev/null +++ b/notebooks/pretrained_dga_model_dsdl.ipynb @@ -0,0 +1,391 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Deep Learning model to detect DGA domains" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This notebook uses a pre-trained deep learning model to predict whether a domain is DGA generated or not." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### What are DGA domains ?" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Adversaries may make use of Domain Generation Algorithms (DGAs) to dynamically identify a destination domain for command and control traffic rather than relying on a list of static IP addresses or domains. This has the advantage of making it much harder for defenders to block, track, or take over the command and control channel, as there potentially could be thousands of domains that malware can check for instructions.\n", + "\n", + "DGAs can take the form of apparently random or \"gibberish\" strings (ex: istgmxdejdnxuyla.ru) when they construct domain names by generating each letter. Alternatively, some DGAs employ whole words as the unit by concatenating words together instead of letters (ex: cityjulydish.net). Many DGAs are time-based, generating a different domain for each time period (hourly, daily, monthly, etc). Others incorporate a seed value as well to make predicting future domains more difficult for defenders https://attack.mitre.org/techniques/T1568/002/" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Stage 0 - import libraries\n", + "At stage 0 we define all imports necessary to run our subsequent code depending on various libraries." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "deletable": false, + "name": "mltkc_import" + }, + "outputs": [], + "source": [ + "import time\n", + "from sklearn.metrics import classification_report, confusion_matrix\n", + "import pandas as pd\n", + "import tensorflow as tf\n", + "from tensorflow.keras import Input \n", + "from tensorflow.keras.models import Sequential\n", + "from tensorflow.keras import layers\n", + "from tensorflow.keras.layers import Embedding\n", + "from tensorflow.keras.preprocessing.text import Tokenizer\n", + "from tensorflow.keras.preprocessing.sequence import pad_sequences\n", + "from tensorflow.keras.layers import Embedding,LSTM,Dropout,Dense,Activation\n", + "from tensorflow.keras.activations import sigmoid\n", + "import matplotlib.pyplot as plt\n", + "import math\n", + "import numpy as np\n", + "import pickle\n", + "from sklearn.utils import shuffle\n", + "from tensorflow.keras.preprocessing import sequence\n", + "import datetime\n", + "from collections import Counter\n", + "MODEL_DIRECTORY = \"/srv/app/model/data/\"\n", + "MAX_VOCAB = 40\n", + "pd.options.mode.chained_assignment = None \n", + "import warnings\n", + "warnings.filterwarnings('ignore')" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes\n", + "print(\"numpy version: \" + np.__version__)\n", + "print(\"pandas version: \" + pd.__version__)\n", + "print(\"tensorflow version: \" + tf.__version__)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Stage 1 - get a data sample from Splunk\n", + "This notebook does not train" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "deletable": false, + "name": "mltkc_stage" + }, + "outputs": [], + "source": [ + "# this cell is not executed from MLTK and should only be used for staging data into the notebook environment\n", + "def stage(name):\n", + " with open(\"data/\"+name+\".csv\", 'r') as f:\n", + " df = pd.read_csv(f)\n", + " with open(\"data/\"+name+\".json\", 'r') as f:\n", + " param = json.load(f)\n", + " return df, param" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes\n", + "df, param = stage(\"pretrained_dga_model_dsdl\")\n", + "print(df.describe())\n", + "print(param)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Stage 2 - create and initialize a model" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "deletable": false, + "name": "mltkc_init" + }, + "outputs": [], + "source": [ + "# initialize your model\n", + "# available inputs: data and parameters\n", + "# returns the model object which will be used as a reference to call fit, apply and summary subsequently\n", + "def init(df,param):\n", + " model = tf.keras.models.load_model(MODEL_DIRECTORY + \"pretrained_dga_model_dsdl\")\n", + " return model" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "scrolled": true + }, + "outputs": [], + "source": [ + "# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes\n", + "print(init(df,param))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Stage 3 - fit the model" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "deletable": false, + "name": "mltkc_fit" + }, + "outputs": [], + "source": [ + "# train your model\n", + "# returns a fit info json object and may modify the model object\n", + "def fit(model,df,param):\n", + " # model.fit()\n", + " info = {\"message\": \"model trained\"}\n", + " return info" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes\n", + "print(fit(model,df,param))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Stage 4 - apply the model" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "deletable": false, + "name": "mltkc_apply" + }, + "outputs": [], + "source": [ + "# apply your model\n", + "# returns the calculated results\n", + "tokenizer = pickle.load(open(MODEL_DIRECTORY+\"pretrained_dga_model_dsdl/tokenizer\", 'rb'))\n", + "vectorizer_words = pickle.load(open(MODEL_DIRECTORY+\"pretrained_dga_model_dsdl/vectorizer_words\", 'rb'))\n", + "vectorizer_non_dga_domains = pickle.load(open(MODEL_DIRECTORY+\"pretrained_dga_model_dsdl/vectorizer_non_dga_domains\", 'rb'))\n", + "weight_words = pickle.load(open(MODEL_DIRECTORY+\"pretrained_dga_model_dsdl/weight_words\", 'rb'))\n", + "weight_non_dga_grams = pickle.load(open(MODEL_DIRECTORY+\"pretrained_dga_model_dsdl/weight_non_dga_grams\", 'rb'))\n", + "domains = pickle.load(open(MODEL_DIRECTORY+\"pretrained_dga_model_dsdl/domains\", 'rb'))\n", + "model = tf.keras.models.load_model(MODEL_DIRECTORY + \"pretrained_dga_model_dsdl\")\n", + "\n", + "def entropy(domain):\n", + " p, lns = Counter(domain), float(len(domain))\n", + " return -sum( count/lns * math.log(count/lns, 2) for count in p.values())\n", + " \n", + "alexa_domains = domains['domain']\n", + "\n", + " \n", + "def is_in_alexa1m(domain,domains):\n", + " test_in_alexa_domains = set(alexa_domains) & domains\n", + " return ((domain in test_in_alexa_domains))\n", + "\n", + " \n", + " \n", + "def add_features(df):\n", + " print (\"1. Done adding ngram features\")\n", + " X_1= weight_words * vectorizer_words.transform(df['domain']).T \n", + " X_2= weight_non_dga_grams * vectorizer_non_dga_domains.transform(df['domain']).T\n", + " X_3 = df['domain'].map(lambda x: entropy(x)) \n", + " print (\"2. Done adding entropy\")\n", + " X_4 = df['domain'].map(lambda x: len(x))\n", + " print (\"3. Done adding length of domain\")\n", + " domains = set(df['domain'])\n", + " X_5 = df['domain'].map(lambda x: is_in_alexa1m(x,domains))\n", + " print (\"4. Done adding domain present in alexa domains\")\n", + " X_5 = X_5.astype(int)\n", + " input2 = np.c_[X_1,X_2,X_3,X_4,X_5] #\n", + " print (\"appending done\")\n", + " return input2 \n", + "\n", + "def prep_text(texts):\n", + " text_sequences = tokenizer.texts_to_sequences(texts)\n", + " return sequence.pad_sequences(text_sequences, maxlen=MAX_VOCAB)\n", + "\n", + "def apply(model,df,param):\n", + " input1 = prep_text(df['domain'])\n", + " input2 = add_features(df)\n", + " \n", + " yhat = model.predict([input1,input2])\n", + " y_ = (yhat > 0.5).astype(\"int32\")\n", + " output = pd.DataFrame()\n", + " #output['domain'] = df['domain']\n", + " #output['true_dga'] = df['is_dga'] \n", + " output['pred_dga']=y_.ravel()\n", + " output['pred_dga_proba']=yhat.ravel()\n", + " return output" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes\n", + "print(apply(model,df,param))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Stage 5 - save the model" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "deletable": false, + "name": "mltkc_save" + }, + "outputs": [], + "source": [ + "# save model to name in expected convention \"_\"\n", + "def save(model,name):\n", + " with open(MODEL_DIRECTORY + name + \".json\", 'w') as file:\n", + " json.dump(model, file)\n", + " return model" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Stage 6 - load the model" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "deletable": false, + "name": "mltkc_load" + }, + "outputs": [], + "source": [ + "# load model from name in expected convention \"_\"\n", + "def load(name):\n", + " model = tf.keras.models.load_model(MODEL_DIRECTORY + \"pretrained_dga_model_dsdl\")\n", + " return model" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Stage 7 - provide a summary of the model" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "deletable": false, + "name": "mltkc_summary" + }, + "outputs": [], + "source": [ + "# return a model summary\n", + "def summary(model=None):\n", + " returns = {\"version\": {\"numpy\": np.__version__, \"pandas\": pd.__version__} }\n", + " return returns" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## End of Stages\n", + "All subsequent cells are not tagged and can be used for further freeform code" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.8.5" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} diff --git a/notebooks/pretrained_dga_model_dsdl.json b/notebooks/pretrained_dga_model_dsdl.json new file mode 100644 index 0000000000..5b026f0a7d --- /dev/null +++ b/notebooks/pretrained_dga_model_dsdl.json @@ -0,0 +1 @@ +{"options": {"params": {"mode": "stage", "algo": "pretrained_dga_model_dsdl"}, "args": ["is_dga", "domain"], "target_variable": ["is_dga"], "feature_variables": ["domain"], "model_name": "pretrained_dga_model_dsdl", "algo_name": "MLTKContainer", "mlspl_limits": {"handle_new_cat": "default", "max_distinct_cat_values": "100", "max_distinct_cat_values_for_classifiers": "100", "max_distinct_cat_values_for_scoring": "100", "max_fit_time": "600", "max_inputs": "100000", "max_memory_usage_mb": "4000", "max_model_size_mb": "30", "max_score_time": "600", "use_sampling": "true"}, "kfold_cv": null}, "feature_variables": ["domain"], "target_variables": ["is_dga"]} \ No newline at end of file