From 4dbc42c86c8743c3a4c82719f5b45c9657e760f2 Mon Sep 17 00:00:00 2001 From: Michael Hart Date: Mon, 28 Jun 2021 22:10:15 -0400 Subject: [PATCH 1/3] Initial commit --- ...oders_for_unusual_group_of_processes.ipynb | 312 ++++++++++++++++++ 1 file changed, 312 insertions(+) create mode 100644 notebooks/autoencoders_for_unusual_group_of_processes.ipynb diff --git a/notebooks/autoencoders_for_unusual_group_of_processes.ipynb b/notebooks/autoencoders_for_unusual_group_of_processes.ipynb new file mode 100644 index 0000000000..bffeb36bd8 --- /dev/null +++ b/notebooks/autoencoders_for_unusual_group_of_processes.ipynb @@ -0,0 +1,312 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "3c1c8020-a12c-49ca-bbe6-4cedd473f48a", + "metadata": {}, + "source": [ + "## AutoEncoders to detect unusual groups of processes\n", + "\n", + "This notebook provides a reference we use for training autoencoders to perform anomaly detection. Autoencoders are neural networks that reconstruct its input by first compressing it into a low dimensional encoding and then decompressing that encoding. These networks can be useful for anomaly detection because unusual data will have poor reconstructions. For cybersecurity, we can leverage anomaly detection to find possible attacks because these will present as unusual compared to normal activity.\n", + "\n", + "For our purposes, we will build an encoder to identify anomalous groups (in mathematical terms, a bag) of processes. We focus on processes with the prefix \\\\device\\Windows since attackers leverage these exececutables to live off the land. We use a technique called feature hashing to project the input (a map of process -> counts) into a vector space (convenient for machine learning).\n" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "243c11af-0f58-4bb7-9bc6-1a0a88df8abe", + "metadata": {}, + "outputs": [], + "source": [ + "import numpy as np\n", + "from sklearn.feature_extraction import FeatureHasher\n", + "import tensorflow as tf\n", + "from sklearn.pipeline import Pipeline\n" + ] + }, + { + "cell_type": "markdown", + "id": "151ade38-3b59-482c-80f4-688f04cb2cf8", + "metadata": {}, + "source": [ + "### Training data\n", + "We will creae a toy dataset. Normal data will consist of a sample of four processes, of which, these four process can occur 0-5 times within a sampling period. We assume independence between the processes. The below code block generates the data." + ] + }, + { + "cell_type": "code", + "execution_count": 38, + "id": "c17980df-e29a-4696-8b35-fa440864306f", + "metadata": {}, + "outputs": [], + "source": [ + "# Let's create some dummy data using processes\n", + "# commonly seen with the prefix C:\\Windows\n", + "num_samples = 10000\n", + "def create_dataset(num_samples=10000):\n", + " data = []\n", + " for i in range(num_samples):\n", + " datum = {'cmd.exe': np.round(np.random.uniform(high=5)),\n", + " 'conhost.exe': np.round(np.random.uniform(high=5)),\n", + " 'svchost.exe': np.round(np.random.uniform(high=5)),\n", + " 'werfault.exe': np.round(np.random.uniform(high=5))}\n", + " data.append(datum)\n", + " return data\n", + "\n", + "training_data = create_dataset()\n", + "test_data = create_dataset()\n" + ] + }, + { + "cell_type": "markdown", + "id": "2e5a163c-7f91-468f-bdb3-5c14a244945f", + "metadata": {}, + "source": [ + "### Transforming the data\n", + "We use a scikit learn pipeline to hash the input into a 16 dimensional vector. An example is shown of what the input and output look like." + ] + }, + { + "cell_type": "code", + "execution_count": 54, + "id": "63ca8129-0e0c-43c2-9ee6-b25c0a83390f", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Input data (process -> count map):\n", + "cmd.exe->3.0, conhost.exe->3.0, svchost.exe->5.0, werfault.exe->0.0\n", + "\n", + "\n", + "Vectorized input (16 dimensional)\n", + "[[-3. 2. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]\n" + ] + } + ], + "source": [ + "pipe = Pipeline([('hasher', FeatureHasher(n_features=16))])\n", + "X = pipe.fit_transform(data)\n", + "\n", + "print(\"Input data (process -> count map):\")\n", + "print(\", \".join([f\"{k}->{v}\" for (k, v) in data[0].items()]))\n", + "print(\"\\n\")\n", + "print(\"Vectorized input (16 dimensional)\")\n", + "print(X[0].todense())" + ] + }, + { + "cell_type": "markdown", + "id": "a941fd31-cb58-407d-b8bc-caea2229bb35", + "metadata": {}, + "source": [ + "## Network\n", + "We build our model using TensorFlow Keras. Since the input is already vectorized, we will stack vanilla dense layers with leaky ReLU activations to compress the input into a four dimensional vector encoding and than decompress back into the original." + ] + }, + { + "cell_type": "code", + "execution_count": 55, + "id": "05f287e6-4f5b-454e-8aa0-2d18f4c07612", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Model: \"model_2\"\n", + "_________________________________________________________________\n", + "Layer (type) Output Shape Param # \n", + "=================================================================\n", + "win_processes_hashed (InputL [(None, 16)] 0 \n", + "_________________________________________________________________\n", + "enc_1 (Dense) (None, 8) 136 \n", + "_________________________________________________________________\n", + "leaky_re_lu_8 (LeakyReLU) (None, 8) 0 \n", + "_________________________________________________________________\n", + "enc_2 (Dense) (None, 4) 36 \n", + "_________________________________________________________________\n", + "leaky_re_lu_9 (LeakyReLU) (None, 4) 0 \n", + "_________________________________________________________________\n", + "dec_1 (Dense) (None, 8) 40 \n", + "_________________________________________________________________\n", + "leaky_re_lu_10 (LeakyReLU) (None, 8) 0 \n", + "_________________________________________________________________\n", + "reconstruction (Dense) (None, 16) 144 \n", + "_________________________________________________________________\n", + "leaky_re_lu_11 (LeakyReLU) (None, 16) 0 \n", + "=================================================================\n", + "Total params: 356\n", + "Trainable params: 356\n", + "Non-trainable params: 0\n", + "_________________________________________________________________\n" + ] + } + ], + "source": [ + "ae_input_layer = tf.keras.layers.Input(shape=(16,), name=\"win_processes_hashed\")\n", + "ae_net = tf.keras.layers.Dense(8, name=\"enc_1\")(ae_input_layer)\n", + "ae_net = tf.keras.layers.LeakyReLU()(ae_net)\n", + "ae_net = tf.keras.layers.Dense(4, name=\"enc_2\")(ae_net)\n", + "ae_net = tf.keras.layers.LeakyReLU()(ae_net)\n", + "ae_net = tf.keras.layers.Dense(8, name=\"dec_1\")(ae_net)\n", + "ae_net = tf.keras.layers.LeakyReLU()(ae_net)\n", + "ae_net = tf.keras.layers.Dense(16, name=\"reconstruction\")(ae_net)\n", + "ae_net = tf.keras.layers.LeakyReLU()(ae_net)\n", + "ae_model = tf.keras.models.Model(ae_input_layer, ae_net)\n", + "ae_model.compile('adam', 'mse', ['mae'])\n", + "ae_model.summary()" + ] + }, + { + "cell_type": "code", + "execution_count": 56, + "id": "056ebac9-a317-4840-af1b-acfb2126f8be", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Epoch 1/10\n", + "1250/1250 [==============================] - 1s 549us/step - loss: 0.7084 - mae: 0.2522\n", + "Epoch 2/10\n", + "1250/1250 [==============================] - 1s 587us/step - loss: 0.0053 - mae: 0.0334\n", + "Epoch 3/10\n", + "1250/1250 [==============================] - 1s 550us/step - loss: 0.0012 - mae: 0.0171\n", + "Epoch 4/10\n", + "1250/1250 [==============================] - 1s 519us/step - loss: 4.5895e-04 - mae: 0.0092\n", + "Epoch 5/10\n", + "1250/1250 [==============================] - 1s 554us/step - loss: 1.9394e-04 - mae: 0.0063\n", + "Epoch 6/10\n", + "1250/1250 [==============================] - 1s 575us/step - loss: 1.0455e-04 - mae: 0.0047\n", + "Epoch 7/10\n", + "1250/1250 [==============================] - 1s 543us/step - loss: 6.1987e-05 - mae: 0.0038\n", + "Epoch 8/10\n", + "1250/1250 [==============================] - 1s 556us/step - loss: 3.1945e-05 - mae: 0.0030\n", + "Epoch 9/10\n", + "1250/1250 [==============================] - 1s 571us/step - loss: 2.4396e-05 - mae: 0.0028\n", + "Epoch 10/10\n", + "1250/1250 [==============================] - 1s 562us/step - loss: 2.3492e-05 - mae: 0.0027\n" + ] + }, + { + "data": { + "text/plain": [ + "" + ] + }, + "execution_count": 56, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Training\n", + "ae_model.fit(X, X.todense(), epochs=10, batch_size=8)" + ] + }, + { + "cell_type": "markdown", + "id": "90dfef32-7da8-40d5-98ca-bf85b98f08ae", + "metadata": {}, + "source": [ + "### Anomaly detection\n", + "We can use euclidean distance to compare how similar the input and reconstruction are. We expect that the distance between reconstruction and input for normal data will be small and large for anomalous data.\n", + "\n", + "First we apply the model and get the mean distance to the test data (which is generated by the same process as the training data). We expect this to be small and it is." + ] + }, + { + "cell_type": "code", + "execution_count": 57, + "id": "01f81d20-aa84-4814-bf4b-ccc7f7e26dbe", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "0.010183748708750814" + ] + }, + "execution_count": 57, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "X_test = pipe.transform(test_data)\n", + "np.average(np.sqrt(np.sum(np.square(X_test - ae_model.predict(X_test)), axis=1)))" + ] + }, + { + "cell_type": "markdown", + "id": "078db0c1-5313-452b-bdf9-984c5c2a7359", + "metadata": {}, + "source": [ + "Now let's apply the model to an unusual command that might be seen with discovery. Notice how much larger the distance between the anomalous reconstruction and the mean normal reconstruction." + ] + }, + { + "cell_type": "code", + "execution_count": 58, + "id": "5b27ab39-e1c3-4847-a90f-be3a03fb4e10", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "5.716755530643157" + ] + }, + "execution_count": 58, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "unusual_command = [{\n", + " 'whoami.exe': 1,\n", + " 'net.exe': 3,\n", + " 'ver.exe': 1,\n", + " 'query.exe': 2,\n", + " 'sc.exe': 5}\n", + "]\n", + "X_u = pipe.transform(unusual_commands)\n", + "np.sqrt(np.sum(np.square(X_u - ae_model.predict(X_u))))" + ] + }, + { + "cell_type": "markdown", + "id": "9adfec39-e05d-417d-9a7a-a80b1e34a538", + "metadata": {}, + "source": [ + "### Summary\n", + "Cybersecurity has long employed anomaly detection to identify unusual activity that may be attributable to cyber attacks. This notebook shows how autoencoders, a deep neural network, can take a map of process counts and identify unusual groups. To accomplish this, we use feature hashing to vectorize the map of process -> counts. We train an autoencoder on the vectorized data. This network is able to identify unusual inputs that may be useful for discovering attacks." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.8.5" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} From 04c4263b560abc4540bd0dec186daf520b5af829 Mon Sep 17 00:00:00 2001 From: Michael Hart Date: Tue, 29 Jun 2021 09:47:37 -0400 Subject: [PATCH 2/3] Adding edits --- ...oders_for_unusual_group_of_processes.ipynb | 25 +++++++++++-------- 1 file changed, 15 insertions(+), 10 deletions(-) diff --git a/notebooks/autoencoders_for_unusual_group_of_processes.ipynb b/notebooks/autoencoders_for_unusual_group_of_processes.ipynb index bffeb36bd8..91ccb4d516 100644 --- a/notebooks/autoencoders_for_unusual_group_of_processes.ipynb +++ b/notebooks/autoencoders_for_unusual_group_of_processes.ipynb @@ -7,9 +7,14 @@ "source": [ "## AutoEncoders to detect unusual groups of processes\n", "\n", - "This notebook provides a reference we use for training autoencoders to perform anomaly detection. Autoencoders are neural networks that reconstruct its input by first compressing it into a low dimensional encoding and then decompressing that encoding. These networks can be useful for anomaly detection because unusual data will have poor reconstructions. For cybersecurity, we can leverage anomaly detection to find possible attacks because these will present as unusual compared to normal activity.\n", + "This notebook provides a reference we use for training [autoencoders](https://en.wikipedia.org/wiki/Autoencoder) to perform anomaly detection. Autoencoders are neural networks that attempt to faithfully reconstruct its input by first compressing it into a low dimensional encoding and then decompressing that encoding. These networks can be useful for anomaly detection because unusual data will have poor reconstructions. For cybersecurity, we can leverage anomaly detection to find possible attacks without having to perform significant feature engineering.\n", "\n", - "For our purposes, we will build an encoder to identify anomalous groups (in mathematical terms, a bag) of processes. We focus on processes with the prefix \\\\device\\Windows since attackers leverage these exececutables to live off the land. We use a technique called feature hashing to project the input (a map of process -> counts) into a vector space (convenient for machine learning).\n" + "
\n", + "\n", + "

Diagram by Michaela Massi, some rights reserved

\n", + "
\n", + "\n", + "For our purposes, we will build an autoencoder to identify anomalous groups of processes. We focus on processes with the prefix \\\\\\\\device\\Windows since attackers leverage these executables to [live off the land](https://conf.splunk.com/files/2019/slides/SEC1375.pdf). We use a technique called [feature hashing](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.FeatureHasher.html) to project the input (a map of process -> counts) into a [vector space](https://en.wikipedia.org/wiki/Vector_space) (convenient for machine learning).\n" ] }, { @@ -31,7 +36,7 @@ "metadata": {}, "source": [ "### Training data\n", - "We will creae a toy dataset. Normal data will consist of a sample of four processes, of which, these four process can occur 0-5 times within a sampling period. We assume independence between the processes. The below code block generates the data." + "We will creae a toy dataset. Normal data will consist of a sample of four processes, of which, these four processes can occur 0-5 times within a sampling period. We assume independence between the processes. The below code block generates the data." ] }, { @@ -64,12 +69,12 @@ "metadata": {}, "source": [ "### Transforming the data\n", - "We use a scikit learn pipeline to hash the input into a 16 dimensional vector. An example is shown of what the input and output look like." + "We use a scikit learn pipeline to feature hash the input into a 16 dimensional vector. An example is shown of what the input and output look like." ] }, { "cell_type": "code", - "execution_count": 54, + "execution_count": 59, "id": "63ca8129-0e0c-43c2-9ee6-b25c0a83390f", "metadata": {}, "outputs": [ @@ -78,7 +83,7 @@ "output_type": "stream", "text": [ "Input data (process -> count map):\n", - "cmd.exe->3.0, conhost.exe->3.0, svchost.exe->5.0, werfault.exe->0.0\n", + "cmd.exe->3, conhost.exe->3, svchost.exe->5, werfault.exe->0\n", "\n", "\n", "Vectorized input (16 dimensional)\n", @@ -91,7 +96,7 @@ "X = pipe.fit_transform(data)\n", "\n", "print(\"Input data (process -> count map):\")\n", - "print(\", \".join([f\"{k}->{v}\" for (k, v) in data[0].items()]))\n", + "print(\", \".join([f\"{k}->{int(v)}\" for (k, v) in data[0].items()]))\n", "print(\"\\n\")\n", "print(\"Vectorized input (16 dimensional)\")\n", "print(X[0].todense())" @@ -214,9 +219,9 @@ "metadata": {}, "source": [ "### Anomaly detection\n", - "We can use euclidean distance to compare how similar the input and reconstruction are. We expect that the distance between reconstruction and input for normal data will be small and large for anomalous data.\n", + "We use euclidean distance as a similarity function between the input and reconstruction. We expect that the distance between reconstruction and input will be small for normal data and large for anomalous data.\n", "\n", - "First we apply the model and get the mean distance to the test data (which is generated by the same process as the training data). We expect this to be small and it is." + "First we apply the model and get the mean distance to the test data (which is generated the same way as the training data). We expect this to be small and it is." ] }, { @@ -246,7 +251,7 @@ "id": "078db0c1-5313-452b-bdf9-984c5c2a7359", "metadata": {}, "source": [ - "Now let's apply the model to an unusual command that might be seen with discovery. Notice how much larger the distance between the anomalous reconstruction and the mean normal reconstruction." + "Now let's apply the model to an unusual command that might be seen with [discovery](https://attack.mitre.org/tactics/TA0007/). Notice how much larger the distance between the anomalous reconstruction and the mean normal reconstruction." ] }, { From 8773f14523254a35d66d178401dc2179dc77c7f7 Mon Sep 17 00:00:00 2001 From: Michael Hart Date: Tue, 29 Jun 2021 14:48:27 -0400 Subject: [PATCH 3/3] More edits --- notebooks/autoencoders_for_unusual_group_of_processes.ipynb | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/notebooks/autoencoders_for_unusual_group_of_processes.ipynb b/notebooks/autoencoders_for_unusual_group_of_processes.ipynb index 91ccb4d516..1f595d1ce8 100644 --- a/notebooks/autoencoders_for_unusual_group_of_processes.ipynb +++ b/notebooks/autoencoders_for_unusual_group_of_processes.ipynb @@ -36,7 +36,7 @@ "metadata": {}, "source": [ "### Training data\n", - "We will creae a toy dataset. Normal data will consist of a sample of four processes, of which, these four processes can occur 0-5 times within a sampling period. We assume independence between the processes. The below code block generates the data." + "We will create a toy dataset that will contain which processes launched and how often during some time window (e.g. hour) correllated on one or more entities (e.g. user and machine, machine). For this demonstration, normal data will consist of a sample of four processes, of which, these four processes can occur 0-5 times within a sampling period. We assume independence between the processes. The below code block generates the data." ] }, { @@ -251,7 +251,7 @@ "id": "078db0c1-5313-452b-bdf9-984c5c2a7359", "metadata": {}, "source": [ - "Now let's apply the model to an unusual command that might be seen with [discovery](https://attack.mitre.org/tactics/TA0007/). Notice how much larger the distance between the anomalous reconstruction and the mean normal reconstruction." + "Now let's apply the model to an unusual command that might be seen with [discovery](https://attack.mitre.org/tactics/TA0007/). Typically, we may see at most one of these processes in a sampling window. Notice how much larger the distance between the anomalous reconstruction and the mean normal reconstruction. Therefore, we can call out this unusual collection of processes in a short period of time to an analyst to get a disposition if this behavior is malicious. We may also call out this activity if there are other secondary or weakly predictive signals related to the same user or device." ] }, { @@ -289,7 +289,7 @@ "metadata": {}, "source": [ "### Summary\n", - "Cybersecurity has long employed anomaly detection to identify unusual activity that may be attributable to cyber attacks. This notebook shows how autoencoders, a deep neural network, can take a map of process counts and identify unusual groups. To accomplish this, we use feature hashing to vectorize the map of process -> counts. We train an autoencoder on the vectorized data. This network is able to identify unusual inputs that may be useful for discovering attacks." + "Cybersecurity has long employed anomaly detection to identify unusual activity that may be attributable to cyber attacks. This notebook shows how autoencoders, a deep neural network, can take a map of process counts during a sampling window and identify unusual groups. To accomplish this, we use feature hashing to vectorize the map of process -> counts. We train an autoencoder on the vectorized data. This network is able to identify unusual inputs that may be useful for discovering attacks." ] } ],