From 944b7a7e82c6fa9b008dcd96dbaabd64c79f1fed Mon Sep 17 00:00:00 2001 From: Kumar Sharad Date: Mon, 23 Jan 2023 17:24:37 +0100 Subject: [PATCH 01/13] Detect Suspicious Filename --- ...lenames_using_pretrained_model_in_dsdl.yml | 87 ++ ...mes_using_pretrained_model_in_dsdl.mlmodel | 2 + ...lenames_using_pretrained_model_in_dsdl.yml | 4 + ...names_using_pretrained_model_in_dsdl.ipynb | 969 ++++++++++++++++++ ...enames_using_pretrained_model_in_dsdl.json | 1 + ...lenames_using_pretrained_model_in_dsdl.yml | 3 + 6 files changed, 1066 insertions(+) create mode 100644 detections/experimental/endpoint/detect_suspicious_filenames_using_pretrained_model_in_dsdl.yml create mode 100644 lookups/__mlspl_detect_suspicious_filenames_using_pretrained_model_in_dsdl.mlmodel create mode 100644 lookups/__mlspl_detect_suspicious_filenames_using_pretrained_model_in_dsdl.yml create mode 100644 notebooks/detect_suspicious_filenames_using_pretrained_model_in_dsdl.ipynb create mode 100644 notebooks/detect_suspicious_filenames_using_pretrained_model_in_dsdl.json create mode 100644 notebooks/detect_suspicious_filenames_using_pretrained_model_in_dsdl.yml diff --git a/detections/experimental/endpoint/detect_suspicious_filenames_using_pretrained_model_in_dsdl.yml b/detections/experimental/endpoint/detect_suspicious_filenames_using_pretrained_model_in_dsdl.yml new file mode 100644 index 0000000000..d814966470 --- /dev/null +++ b/detections/experimental/endpoint/detect_suspicious_filenames_using_pretrained_model_in_dsdl.yml @@ -0,0 +1,87 @@ +name: Detect suspicious filenames using a pretrained model in DSDL +id: a15f8977-ad7d-4669-92ef-b59b97219bf5 +version: 1 +date: 2023-01-23 +author: Abhinav Mishra, Kumar Sharad and Namratha Sreekanta, Splunk +type: Anomaly +datamodel: + - Endpoint +description: The following analytic uses a pre-trained Deep Learning model to predict whether a filename is suspicious or not. Malwares and malicious programs such as ransomware often use tactics, techniques, and procedures (TTPs) such as copying malicious files to the local machine to propagate themselves across the network. A key indicator of compromise is that after a successful execution of the malware, it copies itself as an executable file with a randomly generated filename and places this file in one of the directories. Such techniques are seen in several malwares such as TrickBot. We develop machine learning model that uses a Recurrent Neural Network (RNN) to distinguish between malicious and benign filenames. The model is trained independently and is then made available for download. We use a character level RNN to classify malicious vs. benign filenames. The higher is_malicious_prob, the more likely is the filename to be suspicious. The threshold for flagging a filename as suspicious is set as 0.5. +search: '| tstats `security_content_summariesonly` count min(_time) as firstTime max(_time) as lastTime from datamodel=Endpoint.Processes where Processes.process_path = "*\\Windows\\*" by Processes.parent_process_name Processes.process_name Processes.process Processes.user Processes.dest + | where match(process_name, "\w+.exe") + | apply add:detect_suspicious_filenames_using_pretrained_model_in_dsdl + | where is_malicious_prob > 0.5 + | `drop_dm_object_name(Processs)` + | table process_name, user, dest, is_malicious_prob + | `detect_suspicious_filenames_using_pretrained_model_in_dsdl`' +how_to_implement: + - Steps to deploy detect suspicious filenames model into Splunk App + DSDL. This detection depends on the Splunk app for Data Science and Deep + Learning which can be found here - https://splunkbase.splunk.com/app/4607/ + and the Endpoint datamodel. The detection uses a pre-trained + deep learning model that needs to be deployed in the DSDL app. Follow the steps + for deployment here - + https://github.com/splunk/security_content/wiki/How-to-deploy-pre-trained-Deep-Learning-models-for-ESCU. + - Download the artifacts .tar.gz file from the link - TODO update link here + - Download the detect_suspicious_filenames_using_pretrained_model_in_dsdl.ipynb + Jupyter notebook from https://github.com/splunk/security_content/notebooks + - Login to the Jupyter Lab assigned for detect_suspicious_filenames_using_pretrained_model_in_dsdl container. This container should be listed on Containers page for DSDL app. + - Below steps need to be followed inside Jupyter lab + - Upload the detect_suspicious_filenames_using_pretrained_model_in_dsdl.tar.gz file + into app/model/data path using the upload option in the jupyter notebook. + - Untar the artifact detect_suspicious_filenames_using_pretrained_model_in_dsdl.tar.gz using + tar -xf app/model/data/detect_suspicious_filenames_using_pretrained_model_in_dsdl.tar.gz -C app/model/data + - Upload detect_suspicious_filenames_using_pretrained_model_in_dsdl.pynb into Jupyter lab notebooks folder using the upload option in Jupyter lab + - Save the notebook using the save option in jupyter notebook. + - Upload detect_suspicious_filenames_using_pretrained_model_in_dsdl.json into notebooks/data folder. +known_false_positives: False positives may be present if a suspicious filename is similar to a benign filename. +references: + - https://www.cisa.gov/uscert/ncas/alerts/aa20-302a + - https://www.splunk.com/en_us/blog/security/random-words-on-entropy-and-dns.html +tags: + analytic_story: + - Suspicious Command-Line Executions + asset_type: Endpoint + cis20: + - CIS 8 + - CIS 12 + - CIS 13 + confidence: 90 + context: + - Source:Endpoint + - Stage:Execution + impact: 50 + kill_chain_phases: + - Exploitation + message: The process $process$ is running from an unusual place by $user$ on $dest$ with a filename + that appers to be randomly generated. + mitre_attack_id: + - T1059 + nist: + - PR.DS + - PR.PT + - DE.AE + - DE.CM + observable: + - name: dest + type: Hostname + role: + - Victim + - name: user + type: User + role: + - Victim + product: + - Splunk Enterprise + - Splunk Enterprise Security + - Splunk Cloud + required_fields: + - _time + - Processes.process + - Processes.parent_process_name + - Processes.process_name + - Processes.parent_process + - Processes.user + - Processes.dest + risk_score: 45 + security_domain: Endpoint \ No newline at end of file diff --git a/lookups/__mlspl_detect_suspicious_filenames_using_pretrained_model_in_dsdl.mlmodel b/lookups/__mlspl_detect_suspicious_filenames_using_pretrained_model_in_dsdl.mlmodel new file mode 100644 index 0000000000..aa87ba8d8d --- /dev/null +++ b/lookups/__mlspl_detect_suspicious_filenames_using_pretrained_model_in_dsdl.mlmodel @@ -0,0 +1,2 @@ +algo,model,options +MLTKContainer,"{""__mlspl_type"": [""mltkc.MLTKContainer"", ""MLTKContainer""], ""dict"": {""endpoint_url"": ""https://localhost:56982"", ""out_params"": {""params"": {""mode"": ""stage"", ""algo"": ""detect_suspicious_filenames_using_pretrained_model_in_dsdl""}, ""args"": [""label"", ""text""], ""target_variable"": [""label""], ""feature_variables"": [""text""], ""model_name"": ""detect_suspicious_filenames_using_pretrained_model_in_dsdl"", ""algo_name"": ""MLTKContainer"", ""mlspl_limits"": {""handle_new_cat"": ""default"", ""max_distinct_cat_values"": ""100"", ""max_distinct_cat_values_for_classifiers"": ""100"", ""max_distinct_cat_values_for_scoring"": ""100"", ""max_fit_time"": ""600"", ""max_inputs"": ""100000"", ""max_memory_usage_mb"": ""4000"", ""max_model_size_mb"": ""30"", ""max_score_time"": ""600"", ""use_sampling"": ""true""}, ""kfold_cv"": null}, ""feature_variables"": [""text""], ""target_variable"": ""label""}}","{""params"": {""mode"": ""stage"", ""algo"": ""detect_suspicious_filenames_using_pretrained_model_in_dsdl""}, ""args"": [""label"", ""text""], ""target_variable"": [""label""], ""feature_variables"": [""text""], ""model_name"": ""detect_suspicious_filenames_using_pretrained_model_in_dsdl"", ""algo_name"": ""MLTKContainer"", ""mlspl_limits"": {""handle_new_cat"": ""default"", ""max_distinct_cat_values"": ""100"", ""max_distinct_cat_values_for_classifiers"": ""100"", ""max_distinct_cat_values_for_scoring"": ""100"", ""max_fit_time"": ""600"", ""max_inputs"": ""100000"", ""max_memory_usage_mb"": ""4000"", ""max_model_size_mb"": ""30"", ""max_score_time"": ""600"", ""use_sampling"": ""true""}, ""kfold_cv"": null}" diff --git a/lookups/__mlspl_detect_suspicious_filenames_using_pretrained_model_in_dsdl.yml b/lookups/__mlspl_detect_suspicious_filenames_using_pretrained_model_in_dsdl.yml new file mode 100644 index 0000000000..dc8660dc01 --- /dev/null +++ b/lookups/__mlspl_detect_suspicious_filenames_using_pretrained_model_in_dsdl.yml @@ -0,0 +1,4 @@ +description: Detect a suspicious filename using Pretrained Model in DSDL +filename: __mlspl_detect_suspicious_filenames_using_pretrained_model_in_dsdl.mlmodel +name: __mlspl_detect_suspicious_filenames_using_pretrained_model_in_dsdl +case_sensitive_match: 'false' \ No newline at end of file diff --git a/notebooks/detect_suspicious_filenames_using_pretrained_model_in_dsdl.ipynb b/notebooks/detect_suspicious_filenames_using_pretrained_model_in_dsdl.ipynb new file mode 100644 index 0000000000..1ae6056b5e --- /dev/null +++ b/notebooks/detect_suspicious_filenames_using_pretrained_model_in_dsdl.ipynb @@ -0,0 +1,969 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "tags": [] + }, + "source": [ + "# Deep Learning model to detect suspiciously named files" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This notebook uses a pre-trained Deep Learning model to predict whether a filename is suspicious or not.\n", + "\n", + "Malwares and malicious programs such as ransomware often use tactics, techniques, and procedures (TTPs) such as copying malicious files to the local machine to propagate themselves across the network. A key indicator of compromise is that after a successful execution of the malware, it copies itself as an executable file with a randomly generated filename and places this file in one of the directories. Such techniques are seen in several malwares such as TrickBot. \n", + "\n", + "We develop machine learning model that uses a Recurrent Neural Network (RNN) to distinguish between malicious and benign filenames. The model is trained independently and is then made available for download. We use a character level RNN to classify malicious vs. benign filenames. Higher the `is_malicious_prob`, the more likely is the filename to be suspicious. The threshold for flagging a filename as suspicious is set at `0.5`." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Stage 0 - import libraries\n", + "At stage 0 we define all imports necessary to run our subsequent code depending on various libraries." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": { + "deletable": false, + "name": "mltkc_import" + }, + "outputs": [], + "source": [ + "# This definition exposes all python module imports that should be available in all subsequent commands\n", + "\n", + "import pandas as pd\n", + "import numpy as np\n", + "import random\n", + "import string\n", + "import torch\n", + "import torch.nn as nn\n", + "\n", + "# global constants\n", + "MODEL_DIRECTORY = \"/srv/app/model/data/\"\n", + "\n", + "# model parameters\n", + "all_letters = string.ascii_lowercase\n", + "n_letters = len(all_letters)\n", + "n_hidden = 32\n", + "n_categories = 2\n", + "learning_rate = 5e-4" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "numpy version: 1.23.3\n", + "pandas version: 1.5.0\n" + ] + } + ], + "source": [ + "# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes\n", + "print(\"numpy version: \" + np.__version__)\n", + "print(\"pandas version: \" + pd.__version__)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Stage 1 - get a data sample from Splunk\n", + "There are currently 2 ways to retrieve data from Splunk: Option 1 is to interactively pull data from Splunk into the DSDL Jupyter Lab environment. This is useful when the Splunk REST API is accessible from the Jupyter environment and a valid Splunk auth token is defined in the DSDL app. This option has advantages to quickly experiment with different Splunk SPL queries and further interactively work with the search results in Jupyter.\n", + "\n", + "### Option 1 - pull data from Splunk" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [], + "source": [ + "from dsdlsupport import SplunkSearch as SplunkSearch" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [ + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "07142129edfe4bb6b4159c9fbc7ec0d8", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "VBox(children=(HBox(children=(Textarea(value='| makeresults count=10 \\n| streamstats c as i \\n| eval s = i%3 \\…" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "search = SplunkSearch.SplunkSearch()" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
_timefeature_0feature_1feature_2is
02022-10-12T15:58:22.000+02:000.24281632807105780.24467601440846920.87867884710431111
12022-10-12T15:58:22.000+02:000.4586729397997260.44034866429865360.595190808642655622
22022-10-12T15:58:22.000+02:000.42684995615854860.702591426204890.667612095363438130
32022-10-12T15:58:22.000+02:000.99110854370519520.75376388663426040.842388004064559941
42022-10-12T15:58:22.000+02:000.28641621675342320.42401744471862910.759700436610728552
52022-10-12T15:58:22.000+02:000.94034470710903410.107940037734806540.2277815835550427460
62022-10-12T15:58:22.000+02:000.65414054552093150.76573636429384350.0734438714571297271
72022-10-12T15:58:22.000+02:000.81963268807157870.347312355414032940.855673993937671282
82022-10-12T15:58:22.000+02:000.35216192482039330.047717279754579070.0774232959374785490
92022-10-12T15:58:22.000+02:000.24769726162776350.53931880649179220.7553935945034027101
\n", + "
" + ], + "text/plain": [ + " _time feature_0 feature_1 \\\n", + "0 2022-10-12T15:58:22.000+02:00 0.2428163280710578 0.2446760144084692 \n", + "1 2022-10-12T15:58:22.000+02:00 0.458672939799726 0.4403486642986536 \n", + "2 2022-10-12T15:58:22.000+02:00 0.4268499561585486 0.70259142620489 \n", + "3 2022-10-12T15:58:22.000+02:00 0.9911085437051952 0.7537638866342604 \n", + "4 2022-10-12T15:58:22.000+02:00 0.2864162167534232 0.4240174447186291 \n", + "5 2022-10-12T15:58:22.000+02:00 0.9403447071090341 0.10794003773480654 \n", + "6 2022-10-12T15:58:22.000+02:00 0.6541405455209315 0.7657363642938435 \n", + "7 2022-10-12T15:58:22.000+02:00 0.8196326880715787 0.34731235541403294 \n", + "8 2022-10-12T15:58:22.000+02:00 0.3521619248203933 0.04771727975457907 \n", + "9 2022-10-12T15:58:22.000+02:00 0.2476972616277635 0.5393188064917922 \n", + "\n", + " feature_2 i s \n", + "0 0.878678847104311 1 1 \n", + "1 0.5951908086426556 2 2 \n", + "2 0.6676120953634381 3 0 \n", + "3 0.8423880040645599 4 1 \n", + "4 0.7597004366107285 5 2 \n", + "5 0.22778158355504274 6 0 \n", + "6 0.07344387145712972 7 1 \n", + "7 0.8556739939376712 8 2 \n", + "8 0.07742329593747854 9 0 \n", + "9 0.7553935945034027 10 1 " + ] + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "df = search.as_df()\n", + "df" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Option 2 - push data from Splunk\n", + "In Splunk run a search to pipe a dataset into your notebook environment. You utilize the `mode=stage` flag in the in the `| fit` command to do this. The search results are accessible then as csv file with the same model name that is defined in the `into app:` part of the fit statement. Additionally, meta data is retrieved and accessible as json file. In the same way you can further work with the meta data object as it is exposed in the fit and apply function definitions below in stage 3 and 4." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "| makeresults count=10
\n", + "| streamstats c as i
\n", + "| eval s = i%3
\n", + "| eval feature_{s}=0
\n", + "| foreach feature_* [eval \\<\\\\>=random()/pow(2,31)]
\n", + "| fit MLTKContainer mode=stage algo=barebone_template _time feature_* i into app:barebone_template
" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "After you run this search your data set sample is available as a csv inside the container to develop your model. The name is taken from the into keyword (\"barebone_model\" in the example above) or set to \"default\" if no into keyword is present. This step is intended to work with a subset of your data to create your custom model." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": { + "deletable": false, + "name": "mltkc_stage" + }, + "outputs": [], + "source": [ + "# this cell is not executed from MLTK and should only be used for staging data into the notebook environment\n", + "def stage(name):\n", + " with open(\"data/\"+name+\".csv\", 'r') as f:\n", + " df = pd.read_csv(f)\n", + " with open(\"data/\"+name+\".json\", 'r') as f:\n", + " param = json.load(f)\n", + " return df, param" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": {}, + "outputs": [], + "source": [ + "# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes\n", + "df, param = stage(\"barebone_template\")" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
_timeifeature_1feature_2feature_0
0166556512110.7494230.8619800.063747
1166556512120.3284210.4940550.578063
2166556512130.4565880.1354850.935327
3166556512140.0343840.4572410.043726
4166556512150.3196590.8678640.595515
..................
99516655651219960.1717540.1624970.923288
99616655651219970.5009710.8996590.607057
99716655651219980.2090190.7519870.469225
99816655651219990.1634750.8250840.158101
999166556512110000.2086610.5454870.872765
\n", + "

1000 rows × 5 columns

\n", + "
" + ], + "text/plain": [ + " _time i feature_1 feature_2 feature_0\n", + "0 1665565121 1 0.749423 0.861980 0.063747\n", + "1 1665565121 2 0.328421 0.494055 0.578063\n", + "2 1665565121 3 0.456588 0.135485 0.935327\n", + "3 1665565121 4 0.034384 0.457241 0.043726\n", + "4 1665565121 5 0.319659 0.867864 0.595515\n", + ".. ... ... ... ... ...\n", + "995 1665565121 996 0.171754 0.162497 0.923288\n", + "996 1665565121 997 0.500971 0.899659 0.607057\n", + "997 1665565121 998 0.209019 0.751987 0.469225\n", + "998 1665565121 999 0.163475 0.825084 0.158101\n", + "999 1665565121 1000 0.208661 0.545487 0.872765\n", + "\n", + "[1000 rows x 5 columns]" + ] + }, + "execution_count": 11, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "df" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "{'options': {'params': {'mode': 'stage', 'algo': 'barebone_template'},\n", + " 'args': ['_time', 'feature_*', 'i'],\n", + " 'feature_variables': ['_time', 'feature_*', 'i'],\n", + " 'model_name': 'barebone_template',\n", + " 'algo_name': 'MLTKContainer',\n", + " 'mlspl_limits': {'handle_new_cat': 'default',\n", + " 'max_distinct_cat_values': '100',\n", + " 'max_distinct_cat_values_for_classifiers': '100',\n", + " 'max_distinct_cat_values_for_scoring': '100',\n", + " 'max_fit_time': '600',\n", + " 'max_inputs': '100000',\n", + " 'max_memory_usage_mb': '4000',\n", + " 'max_model_size_mb': '30',\n", + " 'max_score_time': '600',\n", + " 'use_sampling': 'true'},\n", + " 'kfold_cv': None},\n", + " 'feature_variables': ['_time', 'feature_1', 'feature_2', 'feature_0', 'i']}" + ] + }, + "execution_count": 8, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "param" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Stage 2 - create and initialize a model" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": { + "deletable": false, + "name": "mltkc_init" + }, + "outputs": [], + "source": [ + "# initialize your model\n", + "# available inputs: data and parameters\n", + "# returns the model object which will be used as a reference to call fit, apply and summary subsequently\n", + "\n", + "class FilenameClassifier(nn.Module):\n", + " def __init__(self, input_size, hidden_size, output_size):\n", + " super(FilenameClassifier, self).__init__()\n", + " self.hidden_size = hidden_size\n", + " self.i2h = nn.Linear(input_size + hidden_size, hidden_size)\n", + " self.i2o = nn.Linear(input_size + hidden_size, output_size)\n", + " self.softmax = nn.LogSoftmax(dim=1)\n", + "\n", + " def forward(self, input, hidden):\n", + " combined = torch.cat((input, hidden), 1)\n", + " hidden = self.i2h(combined)\n", + " output = self.i2o(combined)\n", + " output = self.softmax(output)\n", + " return output, hidden\n", + "\n", + " def initHidden(self):\n", + " return torch.zeros(1, self.hidden_size)\n", + "\n", + " \n", + "def init(df, param):\n", + " model = FilenameClassifier(n_letters, n_hidden, n_categories)\n", + " model.load_state_dict(torch.load(MODEL_DIRECTORY+'/detect_suspicious_filenames_using_pretrained_model_in_dsdl.pt', map_location=torch.device('cpu')))\n", + " model.eval()\n", + " \n", + " return model\n", + " " + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": { + "scrolled": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{'hyperparameter': 42.0}\n" + ] + } + ], + "source": [ + "# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes\n", + "model = init(df,param)\n", + "print(model)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Stage 3 - fit the model" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "deletable": false, + "name": "mltkc_fit" + }, + "outputs": [], + "source": [ + "# train your model\n", + "# returns a fit info json object and may modify the model object\n", + "def fit(model,df,param):\n", + " # model.fit()\n", + " info = {\"message\": \"model trained\"}\n", + " return info" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{'message': 'model trained'}\n" + ] + } + ], + "source": [ + "# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes\n", + "print(fit(model,df,param))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Stage 4 - apply the model" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "metadata": { + "deletable": false, + "name": "mltkc_apply" + }, + "outputs": [], + "source": [ + "# apply your model\n", + "# returns the calculated results\n", + "\n", + "# Find letter index from all_letters\n", + "def letterToIndex(letter):\n", + " return all_letters.find(letter)\n", + "\n", + "\n", + "# Turn a line into a ,\n", + "# or an array of one-hot letter vectors\n", + "def lineToTensor(line):\n", + " tensor = torch.zeros(len(line), 1, n_letters)\n", + " for li, letter in enumerate(line):\n", + " tensor[li][0][letterToIndex(letter)] = 1\n", + " return tensor\n", + "\n", + "\n", + "def evaluate(model, line_tensor):\n", + " hidden = model.initHidden()\n", + "\n", + " for i in range(line_tensor.size()[0]):\n", + " output, hidden = model(line_tensor[i], hidden)\n", + "\n", + " return output\n", + "\n", + "\n", + "def preprocess_txt(input_line):\n", + " input_line = input_line.lower()\n", + " input_line = ''.join(filter(str.islower, input_line))\n", + " \n", + " return input_line\n", + "\n", + "\n", + "def apply(model, df, param):\n", + " is_malicious_prob_lst = []\n", + " for idx, row in df.iterrows():\n", + " input_line = row['text']\n", + " input_line = preprocess_txt(input_line)\n", + " \n", + " if len(input_line) == 0:\n", + " is_mal_prob = 1\n", + " else:\n", + " with torch.no_grad():\n", + " output = evaluate(model, lineToTensor(input_line))\n", + "\n", + " output = torch.exp(output)\n", + " is_mal_prob = round(output[0][1].item(), 4)\n", + "\n", + " is_malicious_prob_lst.append(is_mal_prob)\n", + " \n", + " output = pd.DataFrame()\n", + " output['is_malicious_prob'] = is_malicious_prob_lst\n", + "\n", + " return output" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " index\n", + "0 0\n", + "1 1\n", + "2 2\n", + "3 3\n", + "4 4\n", + ".. ...\n", + "995 995\n", + "996 996\n", + "997 997\n", + "998 998\n", + "999 999\n", + "\n", + "[1000 rows x 1 columns]\n" + ] + } + ], + "source": [ + "# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes\n", + "print(apply(model,df,param))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Stage 5 - save the model" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "deletable": false, + "name": "mltkc_save" + }, + "outputs": [], + "source": [ + "# save model to name in expected convention \"_\"\n", + "def save(model,name):\n", + " with open(MODEL_DIRECTORY + name + \".json\", 'w') as file:\n", + " json.dump(model, file)\n", + " return model" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Stage 6 - load the model" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": { + "deletable": false, + "name": "mltkc_load" + }, + "outputs": [], + "source": [ + "# load model from name in expected convention \"_\"\n", + "def load(name):\n", + " model = FilenameClassifier(n_letters, n_hidden, n_categories)\n", + " model.load_state_dict(torch.load(MODEL_DIRECTORY+'/detect_suspicious_filenames_using_pretrained_model_in_dsdl.pt', map_location=torch.device('cpu')))\n", + " model.eval()\n", + " return model " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Stage 7 - provide a summary of the model" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "metadata": { + "deletable": false, + "name": "mltkc_summary" + }, + "outputs": [], + "source": [ + "# return a model summary\n", + "def summary(model=None):\n", + " returns = {\"version\": {\"numpy\": np.__version__, \"pandas\": pd.__version__} }\n", + " return returns" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "After implementing your fit, apply, save and load you can train your model:
\n", + "| makeresults count=10
\n", + "| streamstats c as i
\n", + "| eval s = i%3
\n", + "| eval feature_{s}=0
\n", + "| foreach feature_* [eval <<FIELD>>=random()/pow(2,31)]
\n", + "| fit MLTKContainer algo=barebone s from feature_* into app:barebone_model
" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Or apply your model:
\n", + "| makeresults count=10
\n", + "| streamstats c as i
\n", + "| eval s = i%3
\n", + "| eval feature_{s}=0
\n", + "| foreach feature_* [eval <<FIELD>>=random()/pow(2,31)]
\n", + "| apply barebone_model as the_meaning_of_life" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Send data back to Splunk HEC\n", + "When you configured the Splunk HEC Settings in the DSDL app you can easily send back data to an index with [Splunk's HTTP Event Collector (HEC)](https://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector). Read more about data formats and options in the [documentation](https://docs.splunk.com/Documentation/Splunk/latest/Data/FormateventsforHTTPEventCollector#Event_metadata).\n", + "\n", + "### Use cases\n", + "- you want to offload longer running, possibly distributed computations that need to deliver results asynchroneously back into Splunk. \n", + "- you might not want to present results back into the search pipeline after your `| fit` or `| apply` command. \n", + "- you can easily utilize this approach for any logging purposes or other profiling tasks in your ML code so you can actively monitor and analyze your processes.\n", + "\n", + "### Example" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "metadata": {}, + "outputs": [], + "source": [ + "from dsdlsupport import SplunkHEC as SplunkHEC\n", + "hec = SplunkHEC.SplunkHEC()" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "metadata": {}, + "outputs": [], + "source": [ + "# example to send 10 hello world events\n", + "response = hec.send_hello_world(10)" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "HEC endpoint http://host.docker.internal:8088/services/collector/event \n", + "returned with status code 200 \n", + "and response message: {\"text\":\"Success\",\"code\":0}\n" + ] + } + ], + "source": [ + "print(\"HEC endpoint %s \\nreturned with status code %s \\nand response message: %s\" % (response.url, response.status_code, response.text))" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [], + "source": [ + "# example to send a JSON object, e.g. to log some data\n", + "from datetime import datetime\n", + "response = hec.send({'event': {'message': 'operation done', 'log_level': 'INFO' }, 'time': datetime.now().timestamp()})" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "HEC endpoint http://host.docker.internal:8088/services/collector/event \n", + "returned with status code 200 \n", + "and response message: {\"text\":\"Success\",\"code\":0}\n" + ] + } + ], + "source": [ + "print(\"HEC endpoint %s \\nreturned with status code %s \\nand response message: %s\" % (response.url, response.status_code, response.text))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## End of Stages\n", + "All subsequent cells are not tagged and can be used for further freeform code" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.13" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} diff --git a/notebooks/detect_suspicious_filenames_using_pretrained_model_in_dsdl.json b/notebooks/detect_suspicious_filenames_using_pretrained_model_in_dsdl.json new file mode 100644 index 0000000000..2bb9179f65 --- /dev/null +++ b/notebooks/detect_suspicious_filenames_using_pretrained_model_in_dsdl.json @@ -0,0 +1 @@ +{"options": {"params": {"mode": "stage", "algo": "detect_suspicious_filenames_using_pretrained_model_in_dsdl"}, "args": ["label", "text"], "target_variable": ["label"], "feature_variables": ["text"], "model_name": "detect_suspicious_filenames_using_pretrained_model_in_dsdl", "algo_name": "MLTKContainer", "mlspl_limits": {"handle_new_cat": "default", "max_distinct_cat_values": "100", "max_distinct_cat_values_for_classifiers": "100", "max_distinct_cat_values_for_scoring": "100", "max_fit_time": "600", "max_inputs": "100000", "max_memory_usage_mb": "4000", "max_model_size_mb": "30", "max_score_time": "600", "use_sampling": "true"}, "kfold_cv": null}, "feature_variables": ["text"], "target_variables": ["label"]} \ No newline at end of file diff --git a/notebooks/detect_suspicious_filenames_using_pretrained_model_in_dsdl.yml b/notebooks/detect_suspicious_filenames_using_pretrained_model_in_dsdl.yml new file mode 100644 index 0000000000..af2fc64c9b --- /dev/null +++ b/notebooks/detect_suspicious_filenames_using_pretrained_model_in_dsdl.yml @@ -0,0 +1,3 @@ +description: A Jupyter notebook for detecting suspicious filenames using a pre-trained model in DSDL +filename: detect_suspicious_filenames_using_pretrained_model_in_dsdl.ipynb +name: detect_suspicious_filenames_using_pretrained_model_in_dsdl \ No newline at end of file From 709068dcb839886d8537b3f2dd7bf25c1df4648e Mon Sep 17 00:00:00 2001 From: Kumar Sharad Date: Fri, 27 Jan 2023 15:48:17 +0100 Subject: [PATCH 02/13] Detect suspicious DNS text YML update --- ...records_using_pretrained_model_in_dsdl.yml | 25 ++++++++++--------- 1 file changed, 13 insertions(+), 12 deletions(-) diff --git a/detections/experimental/network/detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl.yml b/detections/experimental/network/detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl.yml index c244925543..2ecfe5c278 100644 --- a/detections/experimental/network/detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl.yml +++ b/detections/experimental/network/detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl.yml @@ -35,30 +35,31 @@ search: '| tstats `security_content_summariesonly` count min(_time) as firstTime how_to_implement: 'Steps to deploy detect suspicious DNS TXT records model into Splunk App DSDL. This detection depends on the Splunk app for Data Science and Deep - Learning which can be found here - https://splunkbase.splunk.com/app/4607/ + Learning which can be found here - `https://splunkbase.splunk.com/app/4607/` and the Network Resolution datamodel which can be found here - - https://splunkbase.splunk.com/app/1621/. The detection uses a pre-trained + `https://splunkbase.splunk.com/app/1621/`. The detection uses a pre-trained deep learning model that needs to be deployed in DSDL app. Follow the steps for deployment here - `https://github.com/splunk/security_content/wiki/How-to-deploy-pre-trained-Deep-Learning-models-for-ESCU`.\ - * Download the `artifacts .tar.gz` file from the link - https://seal.splunkresearch.com/detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl.tar.gz - Download the `detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl.ipynb` - Jupyter notebook from https://github.com/splunk/security_content/notebooks\ + * Download the `artifacts .tar.gz` file from the link - `https://seal.splunkresearch.com/detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl.tar.gz`. + + * Download the `detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl.ipynb` + Jupyter notebook from `https://github.com/splunk/security_content/notebooks`.\ * Login to the Jupyter Lab assigned for - detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl container. + `detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl` container. This container should be listed on Containers page for DSDL app.\ - * Below steps need to be followed inside Jupyter lab \ + * Below steps need to be followed inside Jupyter lab.\ - * Upload the detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl.tar.gz file + * Upload the `detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl.tar.gz` file into `app/model/data` path using the upload option in the jupyter notebook.\ - * Untar the artifact detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl.tar.gz using `tar -xf app/model/data/detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl.tar.gz -C app/model/data`\ + * Untar the artifact `detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl.tar.gz` using `tar -xf app/model/data/detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl.tar.gz -C app/model/data`.\ - * Upload detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl.pynb into Jupyter lab notebooks folder using the upload option in Jupyter lab \ + * Upload detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl.ipynb` into Jupyter lab notebooks folder using the upload option in Jupyter lab.\ - * Save the notebook using the save option in jupyter notebook. \ + * Save the notebook using the save option in Jupyter notebook.\ * Upload `detect_suspicious_dns_txt_records_using_pretrained_model_in_dsdl.json` into `notebooks/data` folder.' known_false_positives: False positives may be present if DNS TXT record contents @@ -113,4 +114,4 @@ tags: - DNS.dest - DNS.answer risk_score: 45 - security_domain: network + security_domain: network \ No newline at end of file From 87d5b60b0a36ace559197714b13a2ed33d065f55 Mon Sep 17 00:00:00 2001 From: Kumar Sharad Date: Fri, 27 Jan 2023 17:23:58 +0100 Subject: [PATCH 03/13] Detect suspicious Filename updates --- ...lenames_using_pretrained_model_in_dsdl.yml | 150 +++++++++++------- ...mes_using_pretrained_model_in_dsdl.mlmodel | 2 +- ...names_using_pretrained_model_in_dsdl.ipynb | 10 +- 3 files changed, 97 insertions(+), 65 deletions(-) diff --git a/detections/experimental/endpoint/detect_suspicious_filenames_using_pretrained_model_in_dsdl.yml b/detections/experimental/endpoint/detect_suspicious_filenames_using_pretrained_model_in_dsdl.yml index d814966470..70ba24142c 100644 --- a/detections/experimental/endpoint/detect_suspicious_filenames_using_pretrained_model_in_dsdl.yml +++ b/detections/experimental/endpoint/detect_suspicious_filenames_using_pretrained_model_in_dsdl.yml @@ -5,83 +5,113 @@ date: 2023-01-23 author: Abhinav Mishra, Kumar Sharad and Namratha Sreekanta, Splunk type: Anomaly datamodel: - - Endpoint -description: The following analytic uses a pre-trained Deep Learning model to predict whether a filename is suspicious or not. Malwares and malicious programs such as ransomware often use tactics, techniques, and procedures (TTPs) such as copying malicious files to the local machine to propagate themselves across the network. A key indicator of compromise is that after a successful execution of the malware, it copies itself as an executable file with a randomly generated filename and places this file in one of the directories. Such techniques are seen in several malwares such as TrickBot. We develop machine learning model that uses a Recurrent Neural Network (RNN) to distinguish between malicious and benign filenames. The model is trained independently and is then made available for download. We use a character level RNN to classify malicious vs. benign filenames. The higher is_malicious_prob, the more likely is the filename to be suspicious. The threshold for flagging a filename as suspicious is set as 0.5. -search: '| tstats `security_content_summariesonly` count min(_time) as firstTime max(_time) as lastTime from datamodel=Endpoint.Processes where Processes.process_path = "*\\Windows\\*" by Processes.parent_process_name Processes.process_name Processes.process Processes.user Processes.dest - | where match(process_name, "\w+.exe") - | apply add:detect_suspicious_filenames_using_pretrained_model_in_dsdl - | where is_malicious_prob > 0.5 - | `drop_dm_object_name(Processs)` - | table process_name, user, dest, is_malicious_prob - | `detect_suspicious_filenames_using_pretrained_model_in_dsdl`' -how_to_implement: - - Steps to deploy detect suspicious filenames model into Splunk App +- Endpoint +description: The following analytic uses a pre-trained Deep Learning model to + predict whether a filename is suspicious or not. Malwares and malicious + programs such as ransomware often use tactics, techniques, and procedures + (TTPs) such as copying malicious files to the local machine to propagate + themselves across the network. A key indicator of compromise is that after + a successful execution of the malware, it copies itself as an executable file + with a randomly generated filename and places this file in one of the + directories. Such techniques are seen in several malwares such as TrickBot. + We develop machine learning model that uses a Recurrent Neural Network (RNN) + to distinguish between malicious and benign filenames. The model is trained + independently and is then made available for download. We use a character level + RNN to classify malicious vs. benign filenames. The higher is_malicious_prob, + the more likely is the filename to be suspicious (between [0,1]). The threshold + for flagging a filename as suspicious is set as 0.5. +search: '| tstats `security_content_summariesonly` count min(_time) as firstTime + max(_time) as lastTime from datamodel=Endpoint.Processes + by Processes.process_name Processes.parent_process_name Processes.process + Processes.user Processes.dest + | `drop_dm_object_name(Processes)` + | rename process_name as text + | fields text, parent_process_name, process, user, dest + | apply detect_suspicious_filenames_using_pretrained_model_in_dsdl + | rename predicted_label as is_suspicious_score + | rename text as process_name + | where is_suspicious_score > 0.5 + | `detect_suspicious_filenames_using_pretrained_model_in_dsdl_filter`' + +how_to_implement: 'Steps to deploy detect suspicious filenames model into Splunk App DSDL. This detection depends on the Splunk app for Data Science and Deep - Learning which can be found here - https://splunkbase.splunk.com/app/4607/ + Learning which can be found here - `https://splunkbase.splunk.com/app/4607/` and the Endpoint datamodel. The detection uses a pre-trained deep learning model that needs to be deployed in the DSDL app. Follow the steps - for deployment here - - https://github.com/splunk/security_content/wiki/How-to-deploy-pre-trained-Deep-Learning-models-for-ESCU. - - Download the artifacts .tar.gz file from the link - TODO update link here - - Download the detect_suspicious_filenames_using_pretrained_model_in_dsdl.ipynb - Jupyter notebook from https://github.com/splunk/security_content/notebooks - - Login to the Jupyter Lab assigned for detect_suspicious_filenames_using_pretrained_model_in_dsdl container. This container should be listed on Containers page for DSDL app. - - Below steps need to be followed inside Jupyter lab - - Upload the detect_suspicious_filenames_using_pretrained_model_in_dsdl.tar.gz file - into app/model/data path using the upload option in the jupyter notebook. - - Untar the artifact detect_suspicious_filenames_using_pretrained_model_in_dsdl.tar.gz using - tar -xf app/model/data/detect_suspicious_filenames_using_pretrained_model_in_dsdl.tar.gz -C app/model/data - - Upload detect_suspicious_filenames_using_pretrained_model_in_dsdl.pynb into Jupyter lab notebooks folder using the upload option in Jupyter lab - - Save the notebook using the save option in jupyter notebook. - - Upload detect_suspicious_filenames_using_pretrained_model_in_dsdl.json into notebooks/data folder. -known_false_positives: False positives may be present if a suspicious filename is similar to a benign filename. + for deployment here - `https://github.com/splunk/security_content/wiki/How-to-deploy-pre-trained-Deep-Learning-models-for-ESCU`.\ + + * Download the `artifacts .tar.gz` file from the link - `https://seal.splunkresearch.com/detect_suspicious_filenames_using_pretrained_model_in_dsdl.tar.gz`.\ + + * Download the `detect_suspicious_filenames_using_pretrained_model_in_dsdl.ipynb` + Jupyter notebook from the link - `https://github.com/splunk/security_content/notebooks`.\ + + * Login to the Jupyter Lab assigned for `detect_suspicious_filenames_using_pretrained_model_in_dsdl` + container. This container should be listed on Containers page for DSDL app.\ + + * Follow the steps below inside Jupyter Notebook:\ + + * Upload the `detect_suspicious_filenames_using_pretrained_model_in_dsdl.tar.gz` file + into `app/model/data/detect_suspicious_filenames_using_pretrained_model_in_dsdl/` path using the upload option in the Jupyter Notebook.\ + + * Untar the artifact `detect_suspicious_filenames_using_pretrained_model_in_dsdl.tar.gz` using + `tar -xf app/model/data/detect_suspicious_filenames_using_pretrained_model_in_dsdl.tar.gz -C app/model/data/detect_suspicious_filenames_using_pretrained_model_in_dsdl/`.\ + + * Upload `detect_suspicious_filenames_using_pretrained_model_in_dsdl.ipynb` into the Jupyter Notebooks + folder using the upload option in Jupyter Notebook.\ + + * Save the notebook using the save option in Jupyter Notebook.\ + + * Upload `detect_suspicious_filenames_using_pretrained_model_in_dsdl.json` into `notebooks/data` + folder.' +known_false_positives: False positives may be present if a suspicious filename + is similar to a benign filename. references: - - https://www.cisa.gov/uscert/ncas/alerts/aa20-302a - - https://www.splunk.com/en_us/blog/security/random-words-on-entropy-and-dns.html +- https://www.cisa.gov/uscert/ncas/alerts/aa20-302a +- https://www.splunk.com/en_us/blog/security/random-words-on-entropy-and-dns.html tags: analytic_story: - - Suspicious Command-Line Executions + - Suspicious Command-Line Executions asset_type: Endpoint cis20: - - CIS 8 - - CIS 12 - - CIS 13 + - CIS 8 + - CIS 12 + - CIS 13 confidence: 90 context: - - Source:Endpoint - - Stage:Execution + - Source:Endpoint + - Stage:Execution impact: 50 kill_chain_phases: - - Exploitation + - Exploitation message: The process $process$ is running from an unusual place by $user$ on $dest$ with a filename - that appers to be randomly generated. + that appears to be randomly generated. mitre_attack_id: - - T1059 + - T1059 nist: - - PR.DS - - PR.PT - - DE.AE - - DE.CM + - PR.DS + - PR.PT + - DE.AE + - DE.CM observable: - - name: dest - type: Hostname - role: - - Victim - - name: user - type: User - role: - - Victim + - name: dest + type: Hostname + role: + - Victim + - name: user + type: User + role: + - Victim product: - - Splunk Enterprise - - Splunk Enterprise Security - - Splunk Cloud + - Splunk Enterprise + - Splunk Enterprise Security + - Splunk Cloud required_fields: - - _time - - Processes.process - - Processes.parent_process_name - - Processes.process_name - - Processes.parent_process - - Processes.user - - Processes.dest + - _time + - Processes.process + - Processes.parent_process_name + - Processes.process_name + - Processes.parent_process + - Processes.user + - Processes.dest risk_score: 45 security_domain: Endpoint \ No newline at end of file diff --git a/lookups/__mlspl_detect_suspicious_filenames_using_pretrained_model_in_dsdl.mlmodel b/lookups/__mlspl_detect_suspicious_filenames_using_pretrained_model_in_dsdl.mlmodel index aa87ba8d8d..73ba1206cc 100644 --- a/lookups/__mlspl_detect_suspicious_filenames_using_pretrained_model_in_dsdl.mlmodel +++ b/lookups/__mlspl_detect_suspicious_filenames_using_pretrained_model_in_dsdl.mlmodel @@ -1,2 +1,2 @@ algo,model,options -MLTKContainer,"{""__mlspl_type"": [""mltkc.MLTKContainer"", ""MLTKContainer""], ""dict"": {""endpoint_url"": ""https://localhost:56982"", ""out_params"": {""params"": {""mode"": ""stage"", ""algo"": ""detect_suspicious_filenames_using_pretrained_model_in_dsdl""}, ""args"": [""label"", ""text""], ""target_variable"": [""label""], ""feature_variables"": [""text""], ""model_name"": ""detect_suspicious_filenames_using_pretrained_model_in_dsdl"", ""algo_name"": ""MLTKContainer"", ""mlspl_limits"": {""handle_new_cat"": ""default"", ""max_distinct_cat_values"": ""100"", ""max_distinct_cat_values_for_classifiers"": ""100"", ""max_distinct_cat_values_for_scoring"": ""100"", ""max_fit_time"": ""600"", ""max_inputs"": ""100000"", ""max_memory_usage_mb"": ""4000"", ""max_model_size_mb"": ""30"", ""max_score_time"": ""600"", ""use_sampling"": ""true""}, ""kfold_cv"": null}, ""feature_variables"": [""text""], ""target_variable"": ""label""}}","{""params"": {""mode"": ""stage"", ""algo"": ""detect_suspicious_filenames_using_pretrained_model_in_dsdl""}, ""args"": [""label"", ""text""], ""target_variable"": [""label""], ""feature_variables"": [""text""], ""model_name"": ""detect_suspicious_filenames_using_pretrained_model_in_dsdl"", ""algo_name"": ""MLTKContainer"", ""mlspl_limits"": {""handle_new_cat"": ""default"", ""max_distinct_cat_values"": ""100"", ""max_distinct_cat_values_for_classifiers"": ""100"", ""max_distinct_cat_values_for_scoring"": ""100"", ""max_fit_time"": ""600"", ""max_inputs"": ""100000"", ""max_memory_usage_mb"": ""4000"", ""max_model_size_mb"": ""30"", ""max_score_time"": ""600"", ""use_sampling"": ""true""}, ""kfold_cv"": null}" +MLTKContainer,"{""__mlspl_type"": [""mltkc.MLTKContainer"", ""MLTKContainer""], ""dict"": {""endpoint_url"": ""https://localhost:64407"", ""out_params"": {""params"": {""mode"": ""stage"", ""algo"": ""detect_suspicious_filenames_using_pretrained_model_in_dsdl""}, ""args"": [""label"", ""text""], ""target_variable"": [""label""], ""feature_variables"": [""text""], ""model_name"": ""detect_suspicious_filenames_using_pretrained_model_in_dsdl"", ""algo_name"": ""MLTKContainer"", ""mlspl_limits"": {""handle_new_cat"": ""default"", ""max_distinct_cat_values"": ""100"", ""max_distinct_cat_values_for_classifiers"": ""100"", ""max_distinct_cat_values_for_scoring"": ""100"", ""max_fit_time"": ""600"", ""max_inputs"": ""100000"", ""max_memory_usage_mb"": ""4000"", ""max_model_size_mb"": ""30"", ""max_score_time"": ""600"", ""use_sampling"": ""true""}, ""kfold_cv"": null}, ""feature_variables"": [""text""], ""target_variable"": ""label""}}","{""params"": {""mode"": ""stage"", ""algo"": ""detect_suspicious_filenames_using_pretrained_model_in_dsdl""}, ""args"": [""label"", ""text""], ""target_variable"": [""label""], ""feature_variables"": [""text""], ""model_name"": ""detect_suspicious_filenames_using_pretrained_model_in_dsdl"", ""algo_name"": ""MLTKContainer"", ""mlspl_limits"": {""handle_new_cat"": ""default"", ""max_distinct_cat_values"": ""100"", ""max_distinct_cat_values_for_classifiers"": ""100"", ""max_distinct_cat_values_for_scoring"": ""100"", ""max_fit_time"": ""600"", ""max_inputs"": ""100000"", ""max_memory_usage_mb"": ""4000"", ""max_model_size_mb"": ""30"", ""max_score_time"": ""600"", ""use_sampling"": ""true""}, ""kfold_cv"": null}" diff --git a/notebooks/detect_suspicious_filenames_using_pretrained_model_in_dsdl.ipynb b/notebooks/detect_suspicious_filenames_using_pretrained_model_in_dsdl.ipynb index 1ae6056b5e..68787e98a5 100644 --- a/notebooks/detect_suspicious_filenames_using_pretrained_model_in_dsdl.ipynb +++ b/notebooks/detect_suspicious_filenames_using_pretrained_model_in_dsdl.ipynb @@ -17,7 +17,7 @@ "\n", "Malwares and malicious programs such as ransomware often use tactics, techniques, and procedures (TTPs) such as copying malicious files to the local machine to propagate themselves across the network. A key indicator of compromise is that after a successful execution of the malware, it copies itself as an executable file with a randomly generated filename and places this file in one of the directories. Such techniques are seen in several malwares such as TrickBot. \n", "\n", - "We develop machine learning model that uses a Recurrent Neural Network (RNN) to distinguish between malicious and benign filenames. The model is trained independently and is then made available for download. We use a character level RNN to classify malicious vs. benign filenames. Higher the `is_malicious_prob`, the more likely is the filename to be suspicious. The threshold for flagging a filename as suspicious is set at `0.5`." + "We develop machine learning model that uses a Recurrent Neural Network (RNN) to distinguish between malicious and benign filenames. The model is trained independently and is then made available for download. We use a character level RNN to classify malicious vs. benign filenames. Higher the `is_malicious_prob`, the more likely is the filename to be suspicious (between `[0,1]`). The threshold for flagging a filename as suspicious is set at `0.5`." ] }, { @@ -47,7 +47,7 @@ "import torch.nn as nn\n", "\n", "# global constants\n", - "MODEL_DIRECTORY = \"/srv/app/model/data/\"\n", + "MODEL_DIRECTORY = \"/srv/app/model/data/detect_suspicious_filenames_using_pretrained_model_in_dsdl/detect_suspicious_filenames_using_pretrained_model_in_dsdl.pt\"\n", "\n", "# model parameters\n", "all_letters = string.ascii_lowercase\n", @@ -570,7 +570,7 @@ " \n", "def init(df, param):\n", " model = FilenameClassifier(n_letters, n_hidden, n_categories)\n", - " model.load_state_dict(torch.load(MODEL_DIRECTORY+'/detect_suspicious_filenames_using_pretrained_model_in_dsdl.pt', map_location=torch.device('cpu')))\n", + " model.load_state_dict(torch.load(MODEL_DIRECTORY, map_location=torch.device('cpu')))\n", " model.eval()\n", " \n", " return model\n", @@ -683,7 +683,9 @@ "\n", "\n", "def preprocess_txt(input_line):\n", + " input_line = input_line.split('/')[-1]\n", " input_line = input_line.lower()\n", + " input_line = input_line.replace('.exe','')\n", " input_line = ''.join(filter(str.islower, input_line))\n", " \n", " return input_line\n", @@ -785,7 +787,7 @@ "# load model from name in expected convention \"_\"\n", "def load(name):\n", " model = FilenameClassifier(n_letters, n_hidden, n_categories)\n", - " model.load_state_dict(torch.load(MODEL_DIRECTORY+'/detect_suspicious_filenames_using_pretrained_model_in_dsdl.pt', map_location=torch.device('cpu')))\n", + " model.load_state_dict(torch.load(MODEL_DIRECTORY, map_location=torch.device('cpu')))\n", " model.eval()\n", " return model " ] From e57ab2220d4ed821f4b0def9fe4feff479ee3066 Mon Sep 17 00:00:00 2001 From: Kumar Sharad Date: Mon, 30 Jan 2023 17:07:34 +0100 Subject: [PATCH 04/13] Detect suspicious Filename updates --- ...ct_suspicious_filenames_using_pretrained_model_in_dsdl.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/detections/experimental/endpoint/detect_suspicious_filenames_using_pretrained_model_in_dsdl.yml b/detections/experimental/endpoint/detect_suspicious_filenames_using_pretrained_model_in_dsdl.yml index 70ba24142c..f0cbe11892 100644 --- a/detections/experimental/endpoint/detect_suspicious_filenames_using_pretrained_model_in_dsdl.yml +++ b/detections/experimental/endpoint/detect_suspicious_filenames_using_pretrained_model_in_dsdl.yml @@ -1,7 +1,7 @@ -name: Detect suspicious filenames using a pretrained model in DSDL +name: Detect suspicious processes using a pretrained model in DSDL id: a15f8977-ad7d-4669-92ef-b59b97219bf5 version: 1 -date: 2023-01-23 +date: "2023-01-23" author: Abhinav Mishra, Kumar Sharad and Namratha Sreekanta, Splunk type: Anomaly datamodel: From fe3bd00d31d169f552bd6d87ea2863c49edb9244 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 1 Feb 2023 23:00:37 +0000 Subject: [PATCH 05/13] Bump pytest from 7.2.0 to 7.2.1 Bumps [pytest](https://github.com/pytest-dev/pytest) from 7.2.0 to 7.2.1. - [Release notes](https://github.com/pytest-dev/pytest/releases) - [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst) - [Commits](https://github.com/pytest-dev/pytest/compare/7.2.0...7.2.1) --- updated-dependencies: - dependency-name: pytest dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index e98f1fcf46..cb168fafc8 100644 --- a/requirements.txt +++ b/requirements.txt @@ -7,7 +7,7 @@ mock==4.0.3 psutil==5.9.4 pycvesearch==1.2 pydantic==1.10.4 -pytest==7.2.0 +pytest==7.2.1 PyYAML==5.4.1 questionary==1.10.0 requests==2.28.1 From e9ed13ea3172c8e1b5754198e03603fac150df70 Mon Sep 17 00:00:00 2001 From: Kumar Sharad Date: Tue, 7 Feb 2023 15:31:48 +0100 Subject: [PATCH 06/13] Detection name update --- ...ssnames_using_pretrained_model_in_dsdl.yml | 117 +++ ...mes_using_pretrained_model_in_dsdl.mlmodel | 2 + ...ssnames_using_pretrained_model_in_dsdl.yml | 4 + ...names_using_pretrained_model_in_dsdl.ipynb | 973 ++++++++++++++++++ ...snames_using_pretrained_model_in_dsdl.json | 1 + ...ssnames_using_pretrained_model_in_dsdl.yml | 3 + 6 files changed, 1100 insertions(+) create mode 100644 detections/experimental/endpoint/detect_suspicious_processnames_using_pretrained_model_in_dsdl.yml create mode 100644 lookups/__mlspl_detect_suspicious_processnames_using_pretrained_model_in_dsdl.mlmodel create mode 100644 lookups/__mlspl_detect_suspicious_processnames_using_pretrained_model_in_dsdl.yml create mode 100644 notebooks/detect_suspicious_processnames_using_pretrained_model_in_dsdl.ipynb create mode 100644 notebooks/detect_suspicious_processnames_using_pretrained_model_in_dsdl.json create mode 100644 notebooks/detect_suspicious_processnames_using_pretrained_model_in_dsdl.yml diff --git a/detections/experimental/endpoint/detect_suspicious_processnames_using_pretrained_model_in_dsdl.yml b/detections/experimental/endpoint/detect_suspicious_processnames_using_pretrained_model_in_dsdl.yml new file mode 100644 index 0000000000..b11d50833e --- /dev/null +++ b/detections/experimental/endpoint/detect_suspicious_processnames_using_pretrained_model_in_dsdl.yml @@ -0,0 +1,117 @@ +name: Detect suspicious processes using a pretrained model in DSDL +id: a15f8977-ad7d-4669-92ef-b59b97219bf5 +version: 1 +date: "2023-01-23" +author: Abhinav Mishra, Kumar Sharad and Namratha Sreekanta, Splunk +type: Anomaly +datamodel: +- Endpoint +description: The following analytic uses a pre-trained Deep Learning model to + predict whether a processname is suspicious or not. Malwares and malicious + programs such as ransomware often use tactics, techniques, and procedures + (TTPs) such as copying malicious files to the local machine to propagate + themselves across the network. A key indicator of compromise is that after + a successful execution of the malware, it copies itself as an executable file + with a randomly generated filename and places this file in one of the + directories. Such techniques are seen in several malwares such as TrickBot. + We develop machine learning model that uses a Recurrent Neural Network (RNN) + to distinguish between malicious and benign processnames. The model is trained + independently and is then made available for download. We use a character level + RNN to classify malicious vs. benign processnames. The higher is_malicious_prob, + the more likely is the processname to be suspicious (between [0,1]). The threshold + for flagging a processname as suspicious is set as 0.5. +search: '| tstats `security_content_summariesonly` count min(_time) as firstTime + max(_time) as lastTime from datamodel=Endpoint.Processes + by Processes.process_name Processes.parent_process_name Processes.process + Processes.user Processes.dest + | `drop_dm_object_name(Processes)` + | rename process_name as text + | fields text, parent_process_name, process, user, dest + | apply detect_suspicious_processnames_using_pretrained_model_in_dsdl + | rename predicted_label as is_suspicious_score + | rename text as process_name + | where is_suspicious_score > 0.5 + | `detect_suspicious_processnames_using_pretrained_model_in_dsdl_filter`' + +how_to_implement: 'Steps to deploy detect suspicious processnames model into Splunk App + DSDL. This detection depends on the Splunk app for Data Science and Deep + Learning which can be found here - `https://splunkbase.splunk.com/app/4607/` + and the Endpoint datamodel. The detection uses a pre-trained + deep learning model that needs to be deployed in the DSDL app. Follow the steps + for deployment here - `https://github.com/splunk/security_content/wiki/How-to-deploy-pre-trained-Deep-Learning-models-for-ESCU`.\ + + * Download the `artifacts .tar.gz` file from the link - `https://seal.splunkresearch.com/detect_suspicious_processnames_using_pretrained_model_in_dsdl.tar.gz`.\ + + * Download the `detect_suspicious_processnames_using_pretrained_model_in_dsdl.ipynb` + Jupyter notebook from the link - `https://github.com/splunk/security_content/notebooks`.\ + + * Login to the Jupyter Lab assigned for `detect_suspicious_processnames_using_pretrained_model_in_dsdl` + container. This container should be listed on Containers page for DSDL app.\ + + * Follow the steps below inside Jupyter Notebook:\ + + * Upload the `detect_suspicious_processnames_using_pretrained_model_in_dsdl.tar.gz` file + into `app/model/data/detect_suspicious_processnames_using_pretrained_model_in_dsdl/` path using the upload option in the Jupyter Notebook.\ + + * Untar the artifact `detect_suspicious_processnames_using_pretrained_model_in_dsdl.tar.gz` using + `tar -xf app/model/data/detect_suspicious_processnames_using_pretrained_model_in_dsdl.tar.gz -C app/model/data/detect_suspicious_processnames_using_pretrained_model_in_dsdl/`.\ + + * Upload `detect_suspicious_processnames_using_pretrained_model_in_dsdl.ipynb` into the Jupyter Notebooks + folder using the upload option in Jupyter Notebook.\ + + * Save the notebook using the save option in Jupyter Notebook.\ + + * Upload `detect_suspicious_processnames_using_pretrained_model_in_dsdl.json` into `notebooks/data` + folder.' +known_false_positives: False positives may be present if a suspicious processname + is similar to a benign processname. +references: +- https://www.cisa.gov/uscert/ncas/alerts/aa20-302a +- https://www.splunk.com/en_us/blog/security/random-words-on-entropy-and-dns.html +tags: + analytic_story: + - Suspicious Command-Line Executions + asset_type: Endpoint + cis20: + - CIS 8 + - CIS 12 + - CIS 13 + confidence: 90 + context: + - Source:Endpoint + - Stage:Execution + impact: 50 + kill_chain_phases: + - Exploitation + message: The process $process$ is running from an unusual place by $user$ on $dest$ with a processname + that appears to be randomly generated. + mitre_attack_id: + - T1059 + nist: + - PR.DS + - PR.PT + - DE.AE + - DE.CM + observable: + - name: dest + type: Hostname + role: + - Victim + - name: user + type: User + role: + - Victim + product: + - Splunk Enterprise + - Splunk Enterprise Security + - Splunk Cloud + required_fields: + - _time + - Processes.process + - Processes.parent_process_name + - Processes.process_name + - Processes.parent_process + - Processes.user + - Processes.dest + risk_score: 45 + security_domain: Endpoint \ No newline at end of file diff --git a/lookups/__mlspl_detect_suspicious_processnames_using_pretrained_model_in_dsdl.mlmodel b/lookups/__mlspl_detect_suspicious_processnames_using_pretrained_model_in_dsdl.mlmodel new file mode 100644 index 0000000000..7adfaa2dee --- /dev/null +++ b/lookups/__mlspl_detect_suspicious_processnames_using_pretrained_model_in_dsdl.mlmodel @@ -0,0 +1,2 @@ +algo,model,options +MLTKContainer,"{""__mlspl_type"": [""mltkc.MLTKContainer"", ""MLTKContainer""], ""dict"": {""endpoint_url"": ""https://localhost:58216"", ""out_params"": {""params"": {""mode"": ""stage"", ""algo"": ""detect_suspicious_processnames_using_pretrained_model_in_dsdl""}, ""args"": [""label"", ""text""], ""target_variable"": [""label""], ""feature_variables"": [""text""], ""model_name"": ""detect_suspicious_processnames_using_pretrained_model_in_dsdl"", ""algo_name"": ""MLTKContainer"", ""mlspl_limits"": {""handle_new_cat"": ""default"", ""max_distinct_cat_values"": ""100"", ""max_distinct_cat_values_for_classifiers"": ""100"", ""max_distinct_cat_values_for_scoring"": ""100"", ""max_fit_time"": ""600"", ""max_inputs"": ""100000"", ""max_memory_usage_mb"": ""4000"", ""max_model_size_mb"": ""30"", ""max_score_time"": ""600"", ""use_sampling"": ""true""}, ""kfold_cv"": null}, ""feature_variables"": [""text""], ""target_variable"": ""label""}}","{""params"": {""mode"": ""stage"", ""algo"": ""detect_suspicious_processnames_using_pretrained_model_in_dsdl""}, ""args"": [""label"", ""text""], ""target_variable"": [""label""], ""feature_variables"": [""text""], ""model_name"": ""detect_suspicious_processnames_using_pretrained_model_in_dsdl"", ""algo_name"": ""MLTKContainer"", ""mlspl_limits"": {""handle_new_cat"": ""default"", ""max_distinct_cat_values"": ""100"", ""max_distinct_cat_values_for_classifiers"": ""100"", ""max_distinct_cat_values_for_scoring"": ""100"", ""max_fit_time"": ""600"", ""max_inputs"": ""100000"", ""max_memory_usage_mb"": ""4000"", ""max_model_size_mb"": ""30"", ""max_score_time"": ""600"", ""use_sampling"": ""true""}, ""kfold_cv"": null}" diff --git a/lookups/__mlspl_detect_suspicious_processnames_using_pretrained_model_in_dsdl.yml b/lookups/__mlspl_detect_suspicious_processnames_using_pretrained_model_in_dsdl.yml new file mode 100644 index 0000000000..dfe23d09ca --- /dev/null +++ b/lookups/__mlspl_detect_suspicious_processnames_using_pretrained_model_in_dsdl.yml @@ -0,0 +1,4 @@ +description: Detect a suspicious processname using Pretrained Model in DSDL +filename: __mlspl_detect_suspicious_processnames_using_pretrained_model_in_dsdl.mlmodel +name: __mlspl_detect_suspicious_processnames_using_pretrained_model_in_dsdl +case_sensitive_match: 'false' \ No newline at end of file diff --git a/notebooks/detect_suspicious_processnames_using_pretrained_model_in_dsdl.ipynb b/notebooks/detect_suspicious_processnames_using_pretrained_model_in_dsdl.ipynb new file mode 100644 index 0000000000..4d12d17579 --- /dev/null +++ b/notebooks/detect_suspicious_processnames_using_pretrained_model_in_dsdl.ipynb @@ -0,0 +1,973 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "tags": [] + }, + "source": [ + "# Deep Learning model to detect suspiciously named files" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "tags": [] + }, + "source": [ + "This notebook uses a pre-trained Deep Learning model to predict whether a processname is suspicious or not.\n", + "\n", + "Malwares and malicious programs such as ransomware often use tactics, techniques, and procedures (TTPs) such as copying malicious files to the local machine to propagate themselves across the network. A key indicator of compromise is that after a successful execution of the malware, it copies itself as an executable file with a randomly generated filename and places this file in one of the directories. Such techniques are seen in several malwares such as TrickBot. \n", + "\n", + "We develop machine learning model that uses a Recurrent Neural Network (RNN) to distinguish between malicious and benign processnames. The model is trained independently and is then made available for download. We use a character level RNN to classify malicious vs. benign processnames. Higher the `is_malicious_prob`, the more likely is the processname to be suspicious (between `[0,1]`). The threshold for flagging a processname as suspicious is set at `0.5`." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Stage 0 - import libraries\n", + "At stage 0 we define all imports necessary to run our subsequent code depending on various libraries." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": { + "deletable": false, + "name": "mltkc_import" + }, + "outputs": [], + "source": [ + "# This definition exposes all python module imports that should be available in all subsequent commands\n", + "\n", + "import pandas as pd\n", + "import numpy as np\n", + "import random\n", + "import string\n", + "import torch\n", + "import torch.nn as nn\n", + "\n", + "# global constants\n", + "MODEL_DIRECTORY = \"/srv/app/model/data/detect_suspicious_processnames_using_pretrained_model_in_dsdl/detect_suspicious_processnames_using_pretrained_model_in_dsdl.pt\"\n", + "\n", + "# model parameters\n", + "all_letters = string.ascii_lowercase\n", + "n_letters = len(all_letters)\n", + "n_hidden = 32\n", + "n_categories = 2\n", + "learning_rate = 5e-4" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "numpy version: 1.23.3\n", + "pandas version: 1.5.0\n" + ] + } + ], + "source": [ + "# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes\n", + "print(\"numpy version: \" + np.__version__)\n", + "print(\"pandas version: \" + pd.__version__)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Stage 1 - get a data sample from Splunk\n", + "There are currently 2 ways to retrieve data from Splunk: Option 1 is to interactively pull data from Splunk into the DSDL Jupyter Lab environment. This is useful when the Splunk REST API is accessible from the Jupyter environment and a valid Splunk auth token is defined in the DSDL app. This option has advantages to quickly experiment with different Splunk SPL queries and further interactively work with the search results in Jupyter.\n", + "\n", + "### Option 1 - pull data from Splunk" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [], + "source": [ + "from dsdlsupport import SplunkSearch as SplunkSearch" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [ + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "07142129edfe4bb6b4159c9fbc7ec0d8", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "VBox(children=(HBox(children=(Textarea(value='| makeresults count=10 \\n| streamstats c as i \\n| eval s = i%3 \\…" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "search = SplunkSearch.SplunkSearch()" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
_timefeature_0feature_1feature_2is
02022-10-12T15:58:22.000+02:000.24281632807105780.24467601440846920.87867884710431111
12022-10-12T15:58:22.000+02:000.4586729397997260.44034866429865360.595190808642655622
22022-10-12T15:58:22.000+02:000.42684995615854860.702591426204890.667612095363438130
32022-10-12T15:58:22.000+02:000.99110854370519520.75376388663426040.842388004064559941
42022-10-12T15:58:22.000+02:000.28641621675342320.42401744471862910.759700436610728552
52022-10-12T15:58:22.000+02:000.94034470710903410.107940037734806540.2277815835550427460
62022-10-12T15:58:22.000+02:000.65414054552093150.76573636429384350.0734438714571297271
72022-10-12T15:58:22.000+02:000.81963268807157870.347312355414032940.855673993937671282
82022-10-12T15:58:22.000+02:000.35216192482039330.047717279754579070.0774232959374785490
92022-10-12T15:58:22.000+02:000.24769726162776350.53931880649179220.7553935945034027101
\n", + "
" + ], + "text/plain": [ + " _time feature_0 feature_1 \\\n", + "0 2022-10-12T15:58:22.000+02:00 0.2428163280710578 0.2446760144084692 \n", + "1 2022-10-12T15:58:22.000+02:00 0.458672939799726 0.4403486642986536 \n", + "2 2022-10-12T15:58:22.000+02:00 0.4268499561585486 0.70259142620489 \n", + "3 2022-10-12T15:58:22.000+02:00 0.9911085437051952 0.7537638866342604 \n", + "4 2022-10-12T15:58:22.000+02:00 0.2864162167534232 0.4240174447186291 \n", + "5 2022-10-12T15:58:22.000+02:00 0.9403447071090341 0.10794003773480654 \n", + "6 2022-10-12T15:58:22.000+02:00 0.6541405455209315 0.7657363642938435 \n", + "7 2022-10-12T15:58:22.000+02:00 0.8196326880715787 0.34731235541403294 \n", + "8 2022-10-12T15:58:22.000+02:00 0.3521619248203933 0.04771727975457907 \n", + "9 2022-10-12T15:58:22.000+02:00 0.2476972616277635 0.5393188064917922 \n", + "\n", + " feature_2 i s \n", + "0 0.878678847104311 1 1 \n", + "1 0.5951908086426556 2 2 \n", + "2 0.6676120953634381 3 0 \n", + "3 0.8423880040645599 4 1 \n", + "4 0.7597004366107285 5 2 \n", + "5 0.22778158355504274 6 0 \n", + "6 0.07344387145712972 7 1 \n", + "7 0.8556739939376712 8 2 \n", + "8 0.07742329593747854 9 0 \n", + "9 0.7553935945034027 10 1 " + ] + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "df = search.as_df()\n", + "df" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Option 2 - push data from Splunk\n", + "In Splunk run a search to pipe a dataset into your notebook environment. You utilize the `mode=stage` flag in the in the `| fit` command to do this. The search results are accessible then as csv file with the same model name that is defined in the `into app:` part of the fit statement. Additionally, meta data is retrieved and accessible as json file. In the same way you can further work with the meta data object as it is exposed in the fit and apply function definitions below in stage 3 and 4." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "| makeresults count=10
\n", + "| streamstats c as i
\n", + "| eval s = i%3
\n", + "| eval feature_{s}=0
\n", + "| foreach feature_* [eval \\<\\\\>=random()/pow(2,31)]
\n", + "| fit MLTKContainer mode=stage algo=barebone_template _time feature_* i into app:barebone_template
" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "After you run this search your data set sample is available as a csv inside the container to develop your model. The name is taken from the into keyword (\"barebone_model\" in the example above) or set to \"default\" if no into keyword is present. This step is intended to work with a subset of your data to create your custom model." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": { + "deletable": false, + "name": "mltkc_stage" + }, + "outputs": [], + "source": [ + "# this cell is not executed from MLTK and should only be used for staging data into the notebook environment\n", + "def stage(name):\n", + " with open(\"data/\"+name+\".csv\", 'r') as f:\n", + " df = pd.read_csv(f)\n", + " with open(\"data/\"+name+\".json\", 'r') as f:\n", + " param = json.load(f)\n", + " return df, param" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": {}, + "outputs": [], + "source": [ + "# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes\n", + "df, param = stage(\"barebone_template\")" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
_timeifeature_1feature_2feature_0
0166556512110.7494230.8619800.063747
1166556512120.3284210.4940550.578063
2166556512130.4565880.1354850.935327
3166556512140.0343840.4572410.043726
4166556512150.3196590.8678640.595515
..................
99516655651219960.1717540.1624970.923288
99616655651219970.5009710.8996590.607057
99716655651219980.2090190.7519870.469225
99816655651219990.1634750.8250840.158101
999166556512110000.2086610.5454870.872765
\n", + "

1000 rows × 5 columns

\n", + "
" + ], + "text/plain": [ + " _time i feature_1 feature_2 feature_0\n", + "0 1665565121 1 0.749423 0.861980 0.063747\n", + "1 1665565121 2 0.328421 0.494055 0.578063\n", + "2 1665565121 3 0.456588 0.135485 0.935327\n", + "3 1665565121 4 0.034384 0.457241 0.043726\n", + "4 1665565121 5 0.319659 0.867864 0.595515\n", + ".. ... ... ... ... ...\n", + "995 1665565121 996 0.171754 0.162497 0.923288\n", + "996 1665565121 997 0.500971 0.899659 0.607057\n", + "997 1665565121 998 0.209019 0.751987 0.469225\n", + "998 1665565121 999 0.163475 0.825084 0.158101\n", + "999 1665565121 1000 0.208661 0.545487 0.872765\n", + "\n", + "[1000 rows x 5 columns]" + ] + }, + "execution_count": 11, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "df" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "{'options': {'params': {'mode': 'stage', 'algo': 'barebone_template'},\n", + " 'args': ['_time', 'feature_*', 'i'],\n", + " 'feature_variables': ['_time', 'feature_*', 'i'],\n", + " 'model_name': 'barebone_template',\n", + " 'algo_name': 'MLTKContainer',\n", + " 'mlspl_limits': {'handle_new_cat': 'default',\n", + " 'max_distinct_cat_values': '100',\n", + " 'max_distinct_cat_values_for_classifiers': '100',\n", + " 'max_distinct_cat_values_for_scoring': '100',\n", + " 'max_fit_time': '600',\n", + " 'max_inputs': '100000',\n", + " 'max_memory_usage_mb': '4000',\n", + " 'max_model_size_mb': '30',\n", + " 'max_score_time': '600',\n", + " 'use_sampling': 'true'},\n", + " 'kfold_cv': None},\n", + " 'feature_variables': ['_time', 'feature_1', 'feature_2', 'feature_0', 'i']}" + ] + }, + "execution_count": 8, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "param" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Stage 2 - create and initialize a model" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": { + "deletable": false, + "name": "mltkc_init" + }, + "outputs": [], + "source": [ + "# initialize your model\n", + "# available inputs: data and parameters\n", + "# returns the model object which will be used as a reference to call fit, apply and summary subsequently\n", + "\n", + "class ProcessnameClassifier(nn.Module):\n", + " def __init__(self, input_size, hidden_size, output_size):\n", + " super(ProcessnameClassifier, self).__init__()\n", + " self.hidden_size = hidden_size\n", + " self.i2h = nn.Linear(input_size + hidden_size, hidden_size)\n", + " self.i2o = nn.Linear(input_size + hidden_size, output_size)\n", + " self.softmax = nn.LogSoftmax(dim=1)\n", + "\n", + " def forward(self, input, hidden):\n", + " combined = torch.cat((input, hidden), 1)\n", + " hidden = self.i2h(combined)\n", + " output = self.i2o(combined)\n", + " output = self.softmax(output)\n", + " return output, hidden\n", + "\n", + " def initHidden(self):\n", + " return torch.zeros(1, self.hidden_size)\n", + "\n", + " \n", + "def init(df, param):\n", + " model = ProcessnameClassifier(n_letters, n_hidden, n_categories)\n", + " model.load_state_dict(torch.load(MODEL_DIRECTORY, map_location=torch.device('cpu')))\n", + " model.eval()\n", + " \n", + " return model\n", + " " + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": { + "scrolled": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{'hyperparameter': 42.0}\n" + ] + } + ], + "source": [ + "# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes\n", + "model = init(df,param)\n", + "print(model)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Stage 3 - fit the model" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "deletable": false, + "name": "mltkc_fit" + }, + "outputs": [], + "source": [ + "# train your model\n", + "# returns a fit info json object and may modify the model object\n", + "def fit(model,df,param):\n", + " # model.fit()\n", + " info = {\"message\": \"model trained\"}\n", + " return info" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{'message': 'model trained'}\n" + ] + } + ], + "source": [ + "# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes\n", + "print(fit(model,df,param))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Stage 4 - apply the model" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "metadata": { + "deletable": false, + "name": "mltkc_apply" + }, + "outputs": [], + "source": [ + "# apply your model\n", + "# returns the calculated results\n", + "\n", + "# Find letter index from all_letters\n", + "def letterToIndex(letter):\n", + " return all_letters.find(letter)\n", + "\n", + "\n", + "# Turn a line into a ,\n", + "# or an array of one-hot letter vectors\n", + "def lineToTensor(line):\n", + " tensor = torch.zeros(len(line), 1, n_letters)\n", + " for li, letter in enumerate(line):\n", + " tensor[li][0][letterToIndex(letter)] = 1\n", + " return tensor\n", + "\n", + "\n", + "def evaluate(model, line_tensor):\n", + " hidden = model.initHidden()\n", + "\n", + " for i in range(line_tensor.size()[0]):\n", + " output, hidden = model(line_tensor[i], hidden)\n", + "\n", + " return output\n", + "\n", + "\n", + "def preprocess_txt(input_line):\n", + " input_line = input_line.split('/')[-1]\n", + " input_line = input_line.lower()\n", + " input_line = input_line.replace('.exe','')\n", + " input_line = ''.join(filter(str.islower, input_line))\n", + " \n", + " return input_line\n", + "\n", + "\n", + "def apply(model, df, param):\n", + " is_malicious_prob_lst = []\n", + " for idx, row in df.iterrows():\n", + " input_line = row['text']\n", + " input_line = preprocess_txt(input_line)\n", + " \n", + " if len(input_line) == 0:\n", + " is_mal_prob = 1\n", + " else:\n", + " with torch.no_grad():\n", + " output = evaluate(model, lineToTensor(input_line))\n", + "\n", + " output = torch.exp(output)\n", + " is_mal_prob = round(output[0][1].item(), 4)\n", + "\n", + " is_malicious_prob_lst.append(is_mal_prob)\n", + " \n", + " output = pd.DataFrame()\n", + " output['is_malicious_prob'] = is_malicious_prob_lst\n", + "\n", + " return output" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " index\n", + "0 0\n", + "1 1\n", + "2 2\n", + "3 3\n", + "4 4\n", + ".. ...\n", + "995 995\n", + "996 996\n", + "997 997\n", + "998 998\n", + "999 999\n", + "\n", + "[1000 rows x 1 columns]\n" + ] + } + ], + "source": [ + "# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes\n", + "print(apply(model,df,param))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Stage 5 - save the model" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "deletable": false, + "name": "mltkc_save" + }, + "outputs": [], + "source": [ + "# save model to name in expected convention \"_\"\n", + "def save(model,name):\n", + " with open(MODEL_DIRECTORY + name + \".json\", 'w') as file:\n", + " json.dump(model, file)\n", + " return model" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Stage 6 - load the model" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": { + "deletable": false, + "name": "mltkc_load" + }, + "outputs": [], + "source": [ + "# load model from name in expected convention \"_\"\n", + "def load(name):\n", + " model = ProcessnameClassifier(n_letters, n_hidden, n_categories)\n", + " model.load_state_dict(torch.load(MODEL_DIRECTORY, map_location=torch.device('cpu')))\n", + " model.eval()\n", + " return model " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Stage 7 - provide a summary of the model" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "metadata": { + "deletable": false, + "name": "mltkc_summary" + }, + "outputs": [], + "source": [ + "# return a model summary\n", + "def summary(model=None):\n", + " returns = {\"version\": {\"numpy\": np.__version__, \"pandas\": pd.__version__} }\n", + " return returns" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "After implementing your fit, apply, save and load you can train your model:
\n", + "| makeresults count=10
\n", + "| streamstats c as i
\n", + "| eval s = i%3
\n", + "| eval feature_{s}=0
\n", + "| foreach feature_* [eval <<FIELD>>=random()/pow(2,31)]
\n", + "| fit MLTKContainer algo=barebone s from feature_* into app:barebone_model
" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Or apply your model:
\n", + "| makeresults count=10
\n", + "| streamstats c as i
\n", + "| eval s = i%3
\n", + "| eval feature_{s}=0
\n", + "| foreach feature_* [eval <<FIELD>>=random()/pow(2,31)]
\n", + "| apply barebone_model as the_meaning_of_life" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Send data back to Splunk HEC\n", + "When you configured the Splunk HEC Settings in the DSDL app you can easily send back data to an index with [Splunk's HTTP Event Collector (HEC)](https://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector). Read more about data formats and options in the [documentation](https://docs.splunk.com/Documentation/Splunk/latest/Data/FormateventsforHTTPEventCollector#Event_metadata).\n", + "\n", + "### Use cases\n", + "- you want to offload longer running, possibly distributed computations that need to deliver results asynchroneously back into Splunk. \n", + "- you might not want to present results back into the search pipeline after your `| fit` or `| apply` command. \n", + "- you can easily utilize this approach for any logging purposes or other profiling tasks in your ML code so you can actively monitor and analyze your processes.\n", + "\n", + "### Example" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "metadata": {}, + "outputs": [], + "source": [ + "from dsdlsupport import SplunkHEC as SplunkHEC\n", + "hec = SplunkHEC.SplunkHEC()" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "metadata": {}, + "outputs": [], + "source": [ + "# example to send 10 hello world events\n", + "response = hec.send_hello_world(10)" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "HEC endpoint http://host.docker.internal:8088/services/collector/event \n", + "returned with status code 200 \n", + "and response message: {\"text\":\"Success\",\"code\":0}\n" + ] + } + ], + "source": [ + "print(\"HEC endpoint %s \\nreturned with status code %s \\nand response message: %s\" % (response.url, response.status_code, response.text))" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [], + "source": [ + "# example to send a JSON object, e.g. to log some data\n", + "from datetime import datetime\n", + "response = hec.send({'event': {'message': 'operation done', 'log_level': 'INFO' }, 'time': datetime.now().timestamp()})" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "HEC endpoint http://host.docker.internal:8088/services/collector/event \n", + "returned with status code 200 \n", + "and response message: {\"text\":\"Success\",\"code\":0}\n" + ] + } + ], + "source": [ + "print(\"HEC endpoint %s \\nreturned with status code %s \\nand response message: %s\" % (response.url, response.status_code, response.text))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## End of Stages\n", + "All subsequent cells are not tagged and can be used for further freeform code" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.13" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} diff --git a/notebooks/detect_suspicious_processnames_using_pretrained_model_in_dsdl.json b/notebooks/detect_suspicious_processnames_using_pretrained_model_in_dsdl.json new file mode 100644 index 0000000000..136dc7b634 --- /dev/null +++ b/notebooks/detect_suspicious_processnames_using_pretrained_model_in_dsdl.json @@ -0,0 +1 @@ +{"options": {"params": {"mode": "stage", "algo": "detect_suspicious_processnames_using_pretrained_model_in_dsdl"}, "args": ["label", "text"], "target_variable": ["label"], "feature_variables": ["text"], "model_name": "detect_suspicious_processnames_using_pretrained_model_in_dsdl", "algo_name": "MLTKContainer", "mlspl_limits": {"handle_new_cat": "default", "max_distinct_cat_values": "100", "max_distinct_cat_values_for_classifiers": "100", "max_distinct_cat_values_for_scoring": "100", "max_fit_time": "600", "max_inputs": "100000", "max_memory_usage_mb": "4000", "max_model_size_mb": "30", "max_score_time": "600", "use_sampling": "true"}, "kfold_cv": null}, "feature_variables": ["text"], "target_variables": ["label"]} \ No newline at end of file diff --git a/notebooks/detect_suspicious_processnames_using_pretrained_model_in_dsdl.yml b/notebooks/detect_suspicious_processnames_using_pretrained_model_in_dsdl.yml new file mode 100644 index 0000000000..da680fec70 --- /dev/null +++ b/notebooks/detect_suspicious_processnames_using_pretrained_model_in_dsdl.yml @@ -0,0 +1,3 @@ +description: A Jupyter notebook for detecting suspicious processnames using a pre-trained model in DSDL +filename: detect_suspicious_processnames_using_pretrained_model_in_dsdl.ipynb +name: detect_suspicious_processnames_using_pretrained_model_in_dsdl \ No newline at end of file From 3aa32bfc800e030b980ac7182b9d5278ae5b04f2 Mon Sep 17 00:00:00 2001 From: ksharad-splunk <93270665+ksharad-splunk@users.noreply.github.com> Date: Tue, 7 Feb 2023 15:33:44 +0100 Subject: [PATCH 07/13] Delete detect_suspicious_filenames_using_pretrained_model_in_dsdl.yml --- ...lenames_using_pretrained_model_in_dsdl.yml | 117 ------------------ 1 file changed, 117 deletions(-) delete mode 100644 detections/experimental/endpoint/detect_suspicious_filenames_using_pretrained_model_in_dsdl.yml diff --git a/detections/experimental/endpoint/detect_suspicious_filenames_using_pretrained_model_in_dsdl.yml b/detections/experimental/endpoint/detect_suspicious_filenames_using_pretrained_model_in_dsdl.yml deleted file mode 100644 index f0cbe11892..0000000000 --- a/detections/experimental/endpoint/detect_suspicious_filenames_using_pretrained_model_in_dsdl.yml +++ /dev/null @@ -1,117 +0,0 @@ -name: Detect suspicious processes using a pretrained model in DSDL -id: a15f8977-ad7d-4669-92ef-b59b97219bf5 -version: 1 -date: "2023-01-23" -author: Abhinav Mishra, Kumar Sharad and Namratha Sreekanta, Splunk -type: Anomaly -datamodel: -- Endpoint -description: The following analytic uses a pre-trained Deep Learning model to - predict whether a filename is suspicious or not. Malwares and malicious - programs such as ransomware often use tactics, techniques, and procedures - (TTPs) such as copying malicious files to the local machine to propagate - themselves across the network. A key indicator of compromise is that after - a successful execution of the malware, it copies itself as an executable file - with a randomly generated filename and places this file in one of the - directories. Such techniques are seen in several malwares such as TrickBot. - We develop machine learning model that uses a Recurrent Neural Network (RNN) - to distinguish between malicious and benign filenames. The model is trained - independently and is then made available for download. We use a character level - RNN to classify malicious vs. benign filenames. The higher is_malicious_prob, - the more likely is the filename to be suspicious (between [0,1]). The threshold - for flagging a filename as suspicious is set as 0.5. -search: '| tstats `security_content_summariesonly` count min(_time) as firstTime - max(_time) as lastTime from datamodel=Endpoint.Processes - by Processes.process_name Processes.parent_process_name Processes.process - Processes.user Processes.dest - | `drop_dm_object_name(Processes)` - | rename process_name as text - | fields text, parent_process_name, process, user, dest - | apply detect_suspicious_filenames_using_pretrained_model_in_dsdl - | rename predicted_label as is_suspicious_score - | rename text as process_name - | where is_suspicious_score > 0.5 - | `detect_suspicious_filenames_using_pretrained_model_in_dsdl_filter`' - -how_to_implement: 'Steps to deploy detect suspicious filenames model into Splunk App - DSDL. This detection depends on the Splunk app for Data Science and Deep - Learning which can be found here - `https://splunkbase.splunk.com/app/4607/` - and the Endpoint datamodel. The detection uses a pre-trained - deep learning model that needs to be deployed in the DSDL app. Follow the steps - for deployment here - `https://github.com/splunk/security_content/wiki/How-to-deploy-pre-trained-Deep-Learning-models-for-ESCU`.\ - - * Download the `artifacts .tar.gz` file from the link - `https://seal.splunkresearch.com/detect_suspicious_filenames_using_pretrained_model_in_dsdl.tar.gz`.\ - - * Download the `detect_suspicious_filenames_using_pretrained_model_in_dsdl.ipynb` - Jupyter notebook from the link - `https://github.com/splunk/security_content/notebooks`.\ - - * Login to the Jupyter Lab assigned for `detect_suspicious_filenames_using_pretrained_model_in_dsdl` - container. This container should be listed on Containers page for DSDL app.\ - - * Follow the steps below inside Jupyter Notebook:\ - - * Upload the `detect_suspicious_filenames_using_pretrained_model_in_dsdl.tar.gz` file - into `app/model/data/detect_suspicious_filenames_using_pretrained_model_in_dsdl/` path using the upload option in the Jupyter Notebook.\ - - * Untar the artifact `detect_suspicious_filenames_using_pretrained_model_in_dsdl.tar.gz` using - `tar -xf app/model/data/detect_suspicious_filenames_using_pretrained_model_in_dsdl.tar.gz -C app/model/data/detect_suspicious_filenames_using_pretrained_model_in_dsdl/`.\ - - * Upload `detect_suspicious_filenames_using_pretrained_model_in_dsdl.ipynb` into the Jupyter Notebooks - folder using the upload option in Jupyter Notebook.\ - - * Save the notebook using the save option in Jupyter Notebook.\ - - * Upload `detect_suspicious_filenames_using_pretrained_model_in_dsdl.json` into `notebooks/data` - folder.' -known_false_positives: False positives may be present if a suspicious filename - is similar to a benign filename. -references: -- https://www.cisa.gov/uscert/ncas/alerts/aa20-302a -- https://www.splunk.com/en_us/blog/security/random-words-on-entropy-and-dns.html -tags: - analytic_story: - - Suspicious Command-Line Executions - asset_type: Endpoint - cis20: - - CIS 8 - - CIS 12 - - CIS 13 - confidence: 90 - context: - - Source:Endpoint - - Stage:Execution - impact: 50 - kill_chain_phases: - - Exploitation - message: The process $process$ is running from an unusual place by $user$ on $dest$ with a filename - that appears to be randomly generated. - mitre_attack_id: - - T1059 - nist: - - PR.DS - - PR.PT - - DE.AE - - DE.CM - observable: - - name: dest - type: Hostname - role: - - Victim - - name: user - type: User - role: - - Victim - product: - - Splunk Enterprise - - Splunk Enterprise Security - - Splunk Cloud - required_fields: - - _time - - Processes.process - - Processes.parent_process_name - - Processes.process_name - - Processes.parent_process - - Processes.user - - Processes.dest - risk_score: 45 - security_domain: Endpoint \ No newline at end of file From 918f54262dce3aabbe7288dc283b5c40a5c4af14 Mon Sep 17 00:00:00 2001 From: ksharad-splunk <93270665+ksharad-splunk@users.noreply.github.com> Date: Tue, 7 Feb 2023 15:34:23 +0100 Subject: [PATCH 08/13] Delete __mlspl_detect_suspicious_filenames_using_pretrained_model_in_dsdl.mlmodel --- ..._suspicious_filenames_using_pretrained_model_in_dsdl.mlmodel | 2 -- 1 file changed, 2 deletions(-) delete mode 100644 lookups/__mlspl_detect_suspicious_filenames_using_pretrained_model_in_dsdl.mlmodel diff --git a/lookups/__mlspl_detect_suspicious_filenames_using_pretrained_model_in_dsdl.mlmodel b/lookups/__mlspl_detect_suspicious_filenames_using_pretrained_model_in_dsdl.mlmodel deleted file mode 100644 index 73ba1206cc..0000000000 --- a/lookups/__mlspl_detect_suspicious_filenames_using_pretrained_model_in_dsdl.mlmodel +++ /dev/null @@ -1,2 +0,0 @@ -algo,model,options -MLTKContainer,"{""__mlspl_type"": [""mltkc.MLTKContainer"", ""MLTKContainer""], ""dict"": {""endpoint_url"": ""https://localhost:64407"", ""out_params"": {""params"": {""mode"": ""stage"", ""algo"": ""detect_suspicious_filenames_using_pretrained_model_in_dsdl""}, ""args"": [""label"", ""text""], ""target_variable"": [""label""], ""feature_variables"": [""text""], ""model_name"": ""detect_suspicious_filenames_using_pretrained_model_in_dsdl"", ""algo_name"": ""MLTKContainer"", ""mlspl_limits"": {""handle_new_cat"": ""default"", ""max_distinct_cat_values"": ""100"", ""max_distinct_cat_values_for_classifiers"": ""100"", ""max_distinct_cat_values_for_scoring"": ""100"", ""max_fit_time"": ""600"", ""max_inputs"": ""100000"", ""max_memory_usage_mb"": ""4000"", ""max_model_size_mb"": ""30"", ""max_score_time"": ""600"", ""use_sampling"": ""true""}, ""kfold_cv"": null}, ""feature_variables"": [""text""], ""target_variable"": ""label""}}","{""params"": {""mode"": ""stage"", ""algo"": ""detect_suspicious_filenames_using_pretrained_model_in_dsdl""}, ""args"": [""label"", ""text""], ""target_variable"": [""label""], ""feature_variables"": [""text""], ""model_name"": ""detect_suspicious_filenames_using_pretrained_model_in_dsdl"", ""algo_name"": ""MLTKContainer"", ""mlspl_limits"": {""handle_new_cat"": ""default"", ""max_distinct_cat_values"": ""100"", ""max_distinct_cat_values_for_classifiers"": ""100"", ""max_distinct_cat_values_for_scoring"": ""100"", ""max_fit_time"": ""600"", ""max_inputs"": ""100000"", ""max_memory_usage_mb"": ""4000"", ""max_model_size_mb"": ""30"", ""max_score_time"": ""600"", ""use_sampling"": ""true""}, ""kfold_cv"": null}" From 3a4568bc3d41f8476bd17c6c257690ccc3b03f3a Mon Sep 17 00:00:00 2001 From: ksharad-splunk <93270665+ksharad-splunk@users.noreply.github.com> Date: Tue, 7 Feb 2023 15:34:47 +0100 Subject: [PATCH 09/13] Delete __mlspl_detect_suspicious_filenames_using_pretrained_model_in_dsdl.yml --- ...ct_suspicious_filenames_using_pretrained_model_in_dsdl.yml | 4 ---- 1 file changed, 4 deletions(-) delete mode 100644 lookups/__mlspl_detect_suspicious_filenames_using_pretrained_model_in_dsdl.yml diff --git a/lookups/__mlspl_detect_suspicious_filenames_using_pretrained_model_in_dsdl.yml b/lookups/__mlspl_detect_suspicious_filenames_using_pretrained_model_in_dsdl.yml deleted file mode 100644 index dc8660dc01..0000000000 --- a/lookups/__mlspl_detect_suspicious_filenames_using_pretrained_model_in_dsdl.yml +++ /dev/null @@ -1,4 +0,0 @@ -description: Detect a suspicious filename using Pretrained Model in DSDL -filename: __mlspl_detect_suspicious_filenames_using_pretrained_model_in_dsdl.mlmodel -name: __mlspl_detect_suspicious_filenames_using_pretrained_model_in_dsdl -case_sensitive_match: 'false' \ No newline at end of file From 559d7cae628974d6038a9a933a9fdfd278701b6f Mon Sep 17 00:00:00 2001 From: ksharad-splunk <93270665+ksharad-splunk@users.noreply.github.com> Date: Tue, 7 Feb 2023 15:35:11 +0100 Subject: [PATCH 10/13] Delete detect_suspicious_filenames_using_pretrained_model_in_dsdl.ipynb --- ...names_using_pretrained_model_in_dsdl.ipynb | 971 ------------------ 1 file changed, 971 deletions(-) delete mode 100644 notebooks/detect_suspicious_filenames_using_pretrained_model_in_dsdl.ipynb diff --git a/notebooks/detect_suspicious_filenames_using_pretrained_model_in_dsdl.ipynb b/notebooks/detect_suspicious_filenames_using_pretrained_model_in_dsdl.ipynb deleted file mode 100644 index 68787e98a5..0000000000 --- a/notebooks/detect_suspicious_filenames_using_pretrained_model_in_dsdl.ipynb +++ /dev/null @@ -1,971 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "tags": [] - }, - "source": [ - "# Deep Learning model to detect suspiciously named files" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "This notebook uses a pre-trained Deep Learning model to predict whether a filename is suspicious or not.\n", - "\n", - "Malwares and malicious programs such as ransomware often use tactics, techniques, and procedures (TTPs) such as copying malicious files to the local machine to propagate themselves across the network. A key indicator of compromise is that after a successful execution of the malware, it copies itself as an executable file with a randomly generated filename and places this file in one of the directories. Such techniques are seen in several malwares such as TrickBot. \n", - "\n", - "We develop machine learning model that uses a Recurrent Neural Network (RNN) to distinguish between malicious and benign filenames. The model is trained independently and is then made available for download. We use a character level RNN to classify malicious vs. benign filenames. Higher the `is_malicious_prob`, the more likely is the filename to be suspicious (between `[0,1]`). The threshold for flagging a filename as suspicious is set at `0.5`." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Stage 0 - import libraries\n", - "At stage 0 we define all imports necessary to run our subsequent code depending on various libraries." - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "metadata": { - "deletable": false, - "name": "mltkc_import" - }, - "outputs": [], - "source": [ - "# This definition exposes all python module imports that should be available in all subsequent commands\n", - "\n", - "import pandas as pd\n", - "import numpy as np\n", - "import random\n", - "import string\n", - "import torch\n", - "import torch.nn as nn\n", - "\n", - "# global constants\n", - "MODEL_DIRECTORY = \"/srv/app/model/data/detect_suspicious_filenames_using_pretrained_model_in_dsdl/detect_suspicious_filenames_using_pretrained_model_in_dsdl.pt\"\n", - "\n", - "# model parameters\n", - "all_letters = string.ascii_lowercase\n", - "n_letters = len(all_letters)\n", - "n_hidden = 32\n", - "n_categories = 2\n", - "learning_rate = 5e-4" - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "numpy version: 1.23.3\n", - "pandas version: 1.5.0\n" - ] - } - ], - "source": [ - "# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes\n", - "print(\"numpy version: \" + np.__version__)\n", - "print(\"pandas version: \" + pd.__version__)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Stage 1 - get a data sample from Splunk\n", - "There are currently 2 ways to retrieve data from Splunk: Option 1 is to interactively pull data from Splunk into the DSDL Jupyter Lab environment. This is useful when the Splunk REST API is accessible from the Jupyter environment and a valid Splunk auth token is defined in the DSDL app. This option has advantages to quickly experiment with different Splunk SPL queries and further interactively work with the search results in Jupyter.\n", - "\n", - "### Option 1 - pull data from Splunk" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "metadata": {}, - "outputs": [], - "source": [ - "from dsdlsupport import SplunkSearch as SplunkSearch" - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "metadata": {}, - "outputs": [ - { - "data": { - "application/vnd.jupyter.widget-view+json": { - "model_id": "07142129edfe4bb6b4159c9fbc7ec0d8", - "version_major": 2, - "version_minor": 0 - }, - "text/plain": [ - "VBox(children=(HBox(children=(Textarea(value='| makeresults count=10 \\n| streamstats c as i \\n| eval s = i%3 \\…" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "search = SplunkSearch.SplunkSearch()" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
_timefeature_0feature_1feature_2is
02022-10-12T15:58:22.000+02:000.24281632807105780.24467601440846920.87867884710431111
12022-10-12T15:58:22.000+02:000.4586729397997260.44034866429865360.595190808642655622
22022-10-12T15:58:22.000+02:000.42684995615854860.702591426204890.667612095363438130
32022-10-12T15:58:22.000+02:000.99110854370519520.75376388663426040.842388004064559941
42022-10-12T15:58:22.000+02:000.28641621675342320.42401744471862910.759700436610728552
52022-10-12T15:58:22.000+02:000.94034470710903410.107940037734806540.2277815835550427460
62022-10-12T15:58:22.000+02:000.65414054552093150.76573636429384350.0734438714571297271
72022-10-12T15:58:22.000+02:000.81963268807157870.347312355414032940.855673993937671282
82022-10-12T15:58:22.000+02:000.35216192482039330.047717279754579070.0774232959374785490
92022-10-12T15:58:22.000+02:000.24769726162776350.53931880649179220.7553935945034027101
\n", - "
" - ], - "text/plain": [ - " _time feature_0 feature_1 \\\n", - "0 2022-10-12T15:58:22.000+02:00 0.2428163280710578 0.2446760144084692 \n", - "1 2022-10-12T15:58:22.000+02:00 0.458672939799726 0.4403486642986536 \n", - "2 2022-10-12T15:58:22.000+02:00 0.4268499561585486 0.70259142620489 \n", - "3 2022-10-12T15:58:22.000+02:00 0.9911085437051952 0.7537638866342604 \n", - "4 2022-10-12T15:58:22.000+02:00 0.2864162167534232 0.4240174447186291 \n", - "5 2022-10-12T15:58:22.000+02:00 0.9403447071090341 0.10794003773480654 \n", - "6 2022-10-12T15:58:22.000+02:00 0.6541405455209315 0.7657363642938435 \n", - "7 2022-10-12T15:58:22.000+02:00 0.8196326880715787 0.34731235541403294 \n", - "8 2022-10-12T15:58:22.000+02:00 0.3521619248203933 0.04771727975457907 \n", - "9 2022-10-12T15:58:22.000+02:00 0.2476972616277635 0.5393188064917922 \n", - "\n", - " feature_2 i s \n", - "0 0.878678847104311 1 1 \n", - "1 0.5951908086426556 2 2 \n", - "2 0.6676120953634381 3 0 \n", - "3 0.8423880040645599 4 1 \n", - "4 0.7597004366107285 5 2 \n", - "5 0.22778158355504274 6 0 \n", - "6 0.07344387145712972 7 1 \n", - "7 0.8556739939376712 8 2 \n", - "8 0.07742329593747854 9 0 \n", - "9 0.7553935945034027 10 1 " - ] - }, - "execution_count": 7, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "df = search.as_df()\n", - "df" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Option 2 - push data from Splunk\n", - "In Splunk run a search to pipe a dataset into your notebook environment. You utilize the `mode=stage` flag in the in the `| fit` command to do this. The search results are accessible then as csv file with the same model name that is defined in the `into app:` part of the fit statement. Additionally, meta data is retrieved and accessible as json file. In the same way you can further work with the meta data object as it is exposed in the fit and apply function definitions below in stage 3 and 4." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "| makeresults count=10
\n", - "| streamstats c as i
\n", - "| eval s = i%3
\n", - "| eval feature_{s}=0
\n", - "| foreach feature_* [eval \\<\\\\>=random()/pow(2,31)]
\n", - "| fit MLTKContainer mode=stage algo=barebone_template _time feature_* i into app:barebone_template
" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "After you run this search your data set sample is available as a csv inside the container to develop your model. The name is taken from the into keyword (\"barebone_model\" in the example above) or set to \"default\" if no into keyword is present. This step is intended to work with a subset of your data to create your custom model." - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "deletable": false, - "name": "mltkc_stage" - }, - "outputs": [], - "source": [ - "# this cell is not executed from MLTK and should only be used for staging data into the notebook environment\n", - "def stage(name):\n", - " with open(\"data/\"+name+\".csv\", 'r') as f:\n", - " df = pd.read_csv(f)\n", - " with open(\"data/\"+name+\".json\", 'r') as f:\n", - " param = json.load(f)\n", - " return df, param" - ] - }, - { - "cell_type": "code", - "execution_count": 10, - "metadata": {}, - "outputs": [], - "source": [ - "# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes\n", - "df, param = stage(\"barebone_template\")" - ] - }, - { - "cell_type": "code", - "execution_count": 11, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
_timeifeature_1feature_2feature_0
0166556512110.7494230.8619800.063747
1166556512120.3284210.4940550.578063
2166556512130.4565880.1354850.935327
3166556512140.0343840.4572410.043726
4166556512150.3196590.8678640.595515
..................
99516655651219960.1717540.1624970.923288
99616655651219970.5009710.8996590.607057
99716655651219980.2090190.7519870.469225
99816655651219990.1634750.8250840.158101
999166556512110000.2086610.5454870.872765
\n", - "

1000 rows × 5 columns

\n", - "
" - ], - "text/plain": [ - " _time i feature_1 feature_2 feature_0\n", - "0 1665565121 1 0.749423 0.861980 0.063747\n", - "1 1665565121 2 0.328421 0.494055 0.578063\n", - "2 1665565121 3 0.456588 0.135485 0.935327\n", - "3 1665565121 4 0.034384 0.457241 0.043726\n", - "4 1665565121 5 0.319659 0.867864 0.595515\n", - ".. ... ... ... ... ...\n", - "995 1665565121 996 0.171754 0.162497 0.923288\n", - "996 1665565121 997 0.500971 0.899659 0.607057\n", - "997 1665565121 998 0.209019 0.751987 0.469225\n", - "998 1665565121 999 0.163475 0.825084 0.158101\n", - "999 1665565121 1000 0.208661 0.545487 0.872765\n", - "\n", - "[1000 rows x 5 columns]" - ] - }, - "execution_count": 11, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "df" - ] - }, - { - "cell_type": "code", - "execution_count": 8, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "{'options': {'params': {'mode': 'stage', 'algo': 'barebone_template'},\n", - " 'args': ['_time', 'feature_*', 'i'],\n", - " 'feature_variables': ['_time', 'feature_*', 'i'],\n", - " 'model_name': 'barebone_template',\n", - " 'algo_name': 'MLTKContainer',\n", - " 'mlspl_limits': {'handle_new_cat': 'default',\n", - " 'max_distinct_cat_values': '100',\n", - " 'max_distinct_cat_values_for_classifiers': '100',\n", - " 'max_distinct_cat_values_for_scoring': '100',\n", - " 'max_fit_time': '600',\n", - " 'max_inputs': '100000',\n", - " 'max_memory_usage_mb': '4000',\n", - " 'max_model_size_mb': '30',\n", - " 'max_score_time': '600',\n", - " 'use_sampling': 'true'},\n", - " 'kfold_cv': None},\n", - " 'feature_variables': ['_time', 'feature_1', 'feature_2', 'feature_0', 'i']}" - ] - }, - "execution_count": 8, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "param" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Stage 2 - create and initialize a model" - ] - }, - { - "cell_type": "code", - "execution_count": 12, - "metadata": { - "deletable": false, - "name": "mltkc_init" - }, - "outputs": [], - "source": [ - "# initialize your model\n", - "# available inputs: data and parameters\n", - "# returns the model object which will be used as a reference to call fit, apply and summary subsequently\n", - "\n", - "class FilenameClassifier(nn.Module):\n", - " def __init__(self, input_size, hidden_size, output_size):\n", - " super(FilenameClassifier, self).__init__()\n", - " self.hidden_size = hidden_size\n", - " self.i2h = nn.Linear(input_size + hidden_size, hidden_size)\n", - " self.i2o = nn.Linear(input_size + hidden_size, output_size)\n", - " self.softmax = nn.LogSoftmax(dim=1)\n", - "\n", - " def forward(self, input, hidden):\n", - " combined = torch.cat((input, hidden), 1)\n", - " hidden = self.i2h(combined)\n", - " output = self.i2o(combined)\n", - " output = self.softmax(output)\n", - " return output, hidden\n", - "\n", - " def initHidden(self):\n", - " return torch.zeros(1, self.hidden_size)\n", - "\n", - " \n", - "def init(df, param):\n", - " model = FilenameClassifier(n_letters, n_hidden, n_categories)\n", - " model.load_state_dict(torch.load(MODEL_DIRECTORY, map_location=torch.device('cpu')))\n", - " model.eval()\n", - " \n", - " return model\n", - " " - ] - }, - { - "cell_type": "code", - "execution_count": 13, - "metadata": { - "scrolled": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "{'hyperparameter': 42.0}\n" - ] - } - ], - "source": [ - "# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes\n", - "model = init(df,param)\n", - "print(model)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Stage 3 - fit the model" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "deletable": false, - "name": "mltkc_fit" - }, - "outputs": [], - "source": [ - "# train your model\n", - "# returns a fit info json object and may modify the model object\n", - "def fit(model,df,param):\n", - " # model.fit()\n", - " info = {\"message\": \"model trained\"}\n", - " return info" - ] - }, - { - "cell_type": "code", - "execution_count": 16, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "{'message': 'model trained'}\n" - ] - } - ], - "source": [ - "# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes\n", - "print(fit(model,df,param))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Stage 4 - apply the model" - ] - }, - { - "cell_type": "code", - "execution_count": 17, - "metadata": { - "deletable": false, - "name": "mltkc_apply" - }, - "outputs": [], - "source": [ - "# apply your model\n", - "# returns the calculated results\n", - "\n", - "# Find letter index from all_letters\n", - "def letterToIndex(letter):\n", - " return all_letters.find(letter)\n", - "\n", - "\n", - "# Turn a line into a ,\n", - "# or an array of one-hot letter vectors\n", - "def lineToTensor(line):\n", - " tensor = torch.zeros(len(line), 1, n_letters)\n", - " for li, letter in enumerate(line):\n", - " tensor[li][0][letterToIndex(letter)] = 1\n", - " return tensor\n", - "\n", - "\n", - "def evaluate(model, line_tensor):\n", - " hidden = model.initHidden()\n", - "\n", - " for i in range(line_tensor.size()[0]):\n", - " output, hidden = model(line_tensor[i], hidden)\n", - "\n", - " return output\n", - "\n", - "\n", - "def preprocess_txt(input_line):\n", - " input_line = input_line.split('/')[-1]\n", - " input_line = input_line.lower()\n", - " input_line = input_line.replace('.exe','')\n", - " input_line = ''.join(filter(str.islower, input_line))\n", - " \n", - " return input_line\n", - "\n", - "\n", - "def apply(model, df, param):\n", - " is_malicious_prob_lst = []\n", - " for idx, row in df.iterrows():\n", - " input_line = row['text']\n", - " input_line = preprocess_txt(input_line)\n", - " \n", - " if len(input_line) == 0:\n", - " is_mal_prob = 1\n", - " else:\n", - " with torch.no_grad():\n", - " output = evaluate(model, lineToTensor(input_line))\n", - "\n", - " output = torch.exp(output)\n", - " is_mal_prob = round(output[0][1].item(), 4)\n", - "\n", - " is_malicious_prob_lst.append(is_mal_prob)\n", - " \n", - " output = pd.DataFrame()\n", - " output['is_malicious_prob'] = is_malicious_prob_lst\n", - "\n", - " return output" - ] - }, - { - "cell_type": "code", - "execution_count": 18, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - " index\n", - "0 0\n", - "1 1\n", - "2 2\n", - "3 3\n", - "4 4\n", - ".. ...\n", - "995 995\n", - "996 996\n", - "997 997\n", - "998 998\n", - "999 999\n", - "\n", - "[1000 rows x 1 columns]\n" - ] - } - ], - "source": [ - "# THIS CELL IS NOT EXPORTED - free notebook cell for testing or development purposes\n", - "print(apply(model,df,param))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Stage 5 - save the model" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": { - "deletable": false, - "name": "mltkc_save" - }, - "outputs": [], - "source": [ - "# save model to name in expected convention \"_\"\n", - "def save(model,name):\n", - " with open(MODEL_DIRECTORY + name + \".json\", 'w') as file:\n", - " json.dump(model, file)\n", - " return model" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Stage 6 - load the model" - ] - }, - { - "cell_type": "code", - "execution_count": 16, - "metadata": { - "deletable": false, - "name": "mltkc_load" - }, - "outputs": [], - "source": [ - "# load model from name in expected convention \"_\"\n", - "def load(name):\n", - " model = FilenameClassifier(n_letters, n_hidden, n_categories)\n", - " model.load_state_dict(torch.load(MODEL_DIRECTORY, map_location=torch.device('cpu')))\n", - " model.eval()\n", - " return model " - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Stage 7 - provide a summary of the model" - ] - }, - { - "cell_type": "code", - "execution_count": 17, - "metadata": { - "deletable": false, - "name": "mltkc_summary" - }, - "outputs": [], - "source": [ - "# return a model summary\n", - "def summary(model=None):\n", - " returns = {\"version\": {\"numpy\": np.__version__, \"pandas\": pd.__version__} }\n", - " return returns" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "After implementing your fit, apply, save and load you can train your model:
\n", - "| makeresults count=10
\n", - "| streamstats c as i
\n", - "| eval s = i%3
\n", - "| eval feature_{s}=0
\n", - "| foreach feature_* [eval <<FIELD>>=random()/pow(2,31)]
\n", - "| fit MLTKContainer algo=barebone s from feature_* into app:barebone_model
" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Or apply your model:
\n", - "| makeresults count=10
\n", - "| streamstats c as i
\n", - "| eval s = i%3
\n", - "| eval feature_{s}=0
\n", - "| foreach feature_* [eval <<FIELD>>=random()/pow(2,31)]
\n", - "| apply barebone_model as the_meaning_of_life" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Send data back to Splunk HEC\n", - "When you configured the Splunk HEC Settings in the DSDL app you can easily send back data to an index with [Splunk's HTTP Event Collector (HEC)](https://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector). Read more about data formats and options in the [documentation](https://docs.splunk.com/Documentation/Splunk/latest/Data/FormateventsforHTTPEventCollector#Event_metadata).\n", - "\n", - "### Use cases\n", - "- you want to offload longer running, possibly distributed computations that need to deliver results asynchroneously back into Splunk. \n", - "- you might not want to present results back into the search pipeline after your `| fit` or `| apply` command. \n", - "- you can easily utilize this approach for any logging purposes or other profiling tasks in your ML code so you can actively monitor and analyze your processes.\n", - "\n", - "### Example" - ] - }, - { - "cell_type": "code", - "execution_count": 18, - "metadata": {}, - "outputs": [], - "source": [ - "from dsdlsupport import SplunkHEC as SplunkHEC\n", - "hec = SplunkHEC.SplunkHEC()" - ] - }, - { - "cell_type": "code", - "execution_count": 19, - "metadata": {}, - "outputs": [], - "source": [ - "# example to send 10 hello world events\n", - "response = hec.send_hello_world(10)" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "HEC endpoint http://host.docker.internal:8088/services/collector/event \n", - "returned with status code 200 \n", - "and response message: {\"text\":\"Success\",\"code\":0}\n" - ] - } - ], - "source": [ - "print(\"HEC endpoint %s \\nreturned with status code %s \\nand response message: %s\" % (response.url, response.status_code, response.text))" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [], - "source": [ - "# example to send a JSON object, e.g. to log some data\n", - "from datetime import datetime\n", - "response = hec.send({'event': {'message': 'operation done', 'log_level': 'INFO' }, 'time': datetime.now().timestamp()})" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "HEC endpoint http://host.docker.internal:8088/services/collector/event \n", - "returned with status code 200 \n", - "and response message: {\"text\":\"Success\",\"code\":0}\n" - ] - } - ], - "source": [ - "print(\"HEC endpoint %s \\nreturned with status code %s \\nand response message: %s\" % (response.url, response.status_code, response.text))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## End of Stages\n", - "All subsequent cells are not tagged and can be used for further freeform code" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.13" - } - }, - "nbformat": 4, - "nbformat_minor": 4 -} From 22560697cfc50042a9b7efad0cd2bed2e7ce3d42 Mon Sep 17 00:00:00 2001 From: ksharad-splunk <93270665+ksharad-splunk@users.noreply.github.com> Date: Tue, 7 Feb 2023 15:35:31 +0100 Subject: [PATCH 11/13] Delete detect_suspicious_filenames_using_pretrained_model_in_dsdl.json --- ...tect_suspicious_filenames_using_pretrained_model_in_dsdl.json | 1 - 1 file changed, 1 deletion(-) delete mode 100644 notebooks/detect_suspicious_filenames_using_pretrained_model_in_dsdl.json diff --git a/notebooks/detect_suspicious_filenames_using_pretrained_model_in_dsdl.json b/notebooks/detect_suspicious_filenames_using_pretrained_model_in_dsdl.json deleted file mode 100644 index 2bb9179f65..0000000000 --- a/notebooks/detect_suspicious_filenames_using_pretrained_model_in_dsdl.json +++ /dev/null @@ -1 +0,0 @@ -{"options": {"params": {"mode": "stage", "algo": "detect_suspicious_filenames_using_pretrained_model_in_dsdl"}, "args": ["label", "text"], "target_variable": ["label"], "feature_variables": ["text"], "model_name": "detect_suspicious_filenames_using_pretrained_model_in_dsdl", "algo_name": "MLTKContainer", "mlspl_limits": {"handle_new_cat": "default", "max_distinct_cat_values": "100", "max_distinct_cat_values_for_classifiers": "100", "max_distinct_cat_values_for_scoring": "100", "max_fit_time": "600", "max_inputs": "100000", "max_memory_usage_mb": "4000", "max_model_size_mb": "30", "max_score_time": "600", "use_sampling": "true"}, "kfold_cv": null}, "feature_variables": ["text"], "target_variables": ["label"]} \ No newline at end of file From 58476fa5fee1d90039a0b6d086ea03ec3ff0a290 Mon Sep 17 00:00:00 2001 From: ksharad-splunk <93270665+ksharad-splunk@users.noreply.github.com> Date: Tue, 7 Feb 2023 15:35:43 +0100 Subject: [PATCH 12/13] Delete detect_suspicious_filenames_using_pretrained_model_in_dsdl.yml --- ...ect_suspicious_filenames_using_pretrained_model_in_dsdl.yml | 3 --- 1 file changed, 3 deletions(-) delete mode 100644 notebooks/detect_suspicious_filenames_using_pretrained_model_in_dsdl.yml diff --git a/notebooks/detect_suspicious_filenames_using_pretrained_model_in_dsdl.yml b/notebooks/detect_suspicious_filenames_using_pretrained_model_in_dsdl.yml deleted file mode 100644 index af2fc64c9b..0000000000 --- a/notebooks/detect_suspicious_filenames_using_pretrained_model_in_dsdl.yml +++ /dev/null @@ -1,3 +0,0 @@ -description: A Jupyter notebook for detecting suspicious filenames using a pre-trained model in DSDL -filename: detect_suspicious_filenames_using_pretrained_model_in_dsdl.ipynb -name: detect_suspicious_filenames_using_pretrained_model_in_dsdl \ No newline at end of file From aaf4aa05529e75482883ca78bfe01b652307228e Mon Sep 17 00:00:00 2001 From: Kumar Sharad Date: Tue, 7 Feb 2023 18:42:51 +0100 Subject: [PATCH 13/13] Detection name update --- ...t_suspicious_processnames_using_pretrained_model_in_dsdl.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/detections/experimental/endpoint/detect_suspicious_processnames_using_pretrained_model_in_dsdl.yml b/detections/experimental/endpoint/detect_suspicious_processnames_using_pretrained_model_in_dsdl.yml index b11d50833e..fbc9291da5 100644 --- a/detections/experimental/endpoint/detect_suspicious_processnames_using_pretrained_model_in_dsdl.yml +++ b/detections/experimental/endpoint/detect_suspicious_processnames_using_pretrained_model_in_dsdl.yml @@ -1,4 +1,4 @@ -name: Detect suspicious processes using a pretrained model in DSDL +name: Detect suspicious processnames using a pretrained model in DSDL id: a15f8977-ad7d-4669-92ef-b59b97219bf5 version: 1 date: "2023-01-23"