Stand up a 100% containerized Elastic stack, TLS secured, with Elasticsearch, Kibana, Fleet, and the Detection Engine all pre-configured, enabled and ready to use, within minutes.

If you're interested in more details regarding this project and what to do once you have it running, check out our blog post on the Elastic Security Labs site.

⚠️ This is not an Elastic created, sponsored, or maintained project. Elastic is not responsible for this projects design or implementation.

Steps

Git clone this repo
Install prerequisites (see below)
Change into the elastic-container/ folder
Change the default password of changeme in the .env file (don't change the elastic username, it's a required built-in user)
Bulk enable pre-built detection rules by OS in the .env file (not required, see usage below)
Create a Python virtual environment: python -m venv venv
Activate the virtual environment and install dependencies: source venv/bin/activate && pip install -r requirements.txt
Execute the Python script with the start command: python elastic-container.py start
Wait for the prompt to tell you to browse to https://localhost:5601

Requirements

Operating System:

Linux (Ubuntu, Debian, Rocky, Fedora, RHEL, CentOS)
MacOS
Windows (via WSL2)

Prerequisites:

Python 3.10 or higher (required)
Docker suite with docker compose plugin
Git

MacOS:

# Install Homebrew if not already installed
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

# Install prerequisites
brew install python@3.11 git
brew install --cask docker

Once we have Docker installed we need to provide it with privileged access for it to function. Run the following command to open the Docker app and follow the proceeding steps.

open /Applications/Docker.app

Confirm you would like to open the app
Select ok when prompted to provide Docker with privileged access
Enter your password
Close or minimize the Docker app

Ubuntu/Debian:
Please follow the Docker installation instructions. Of specific note, you must install the docker-compose-plugin, which is different than docker-compose.

# Install software-properties-common for PPA support
sudo apt-get update
sudo apt-get install -y software-properties-common

# Add Python 3.11 repository (deadsnakes PPA)
sudo add-apt-repository ppa:deadsnakes/ppa -y
sudo apt-get update

# Install Python 3.11 and tools
sudo apt-get install -y python3.11 python3.11-venv python3.11-distutils python3-pip git

# Install Docker
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
sudo apt-get install -y docker-compose-plugin

# Add user to docker group and start Docker
sudo usermod -aG docker $USER
newgrp docker
sudo systemctl start docker
sudo systemctl enable docker

RPM distributions (Rocky/RHEL/CentOS/Fedora):
Please follow the Docker installation instructions. Of specific note, you must install the docker-compose-plugin, which is different than docker-compose.

# Install Python and tools
sudo dnf install -y python3.11 python3-pip git dnf-plugins-core

# Add Docker's official repository
sudo dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

# Install Docker Engine and Compose plugin
sudo dnf install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

# Add user to docker group and start Docker
sudo usermod -aG docker $USER
newgrp docker
sudo systemctl start docker
sudo systemctl enable docker

Windows 10/11 with WSL 2 (Ubuntu 20.04):
Make sure you are using WSL version 2. You can check the version using wsl -l -v in PowerShell. If the version is wrong you can change it with wsl --set-version Ubuntu-20.04 2

sudo apt-get update

# Follow Ubuntu installation instructions above
sudo apt-get install -y software-properties-common
sudo add-apt-repository ppa:deadsnakes/ppa -y
sudo apt-get update
sudo apt-get install -y python3.11 python3.11-venv python3.11-distutils python3-pip git

Please follow the Docker installation instructions. Of specific note, you must install the docker-compose-plugin, which is different than docker-compose.

Once the Docker suite is installed run sudo service docker start to start it.

Usage

This uses default creds of elastic:changeme and is intended purely for security research on a local Elastic stack. Change the password in the .env file. Don't change the elastic username, it's a required built-in user

This should not be Internet exposed or used in a production environment.

Usage: elastic-container.py [OPTIONS] COMMAND [ARGS]...

Elastic Container - Manage your containerized Elastic Stack

Options:
  -v, --verbose  Enable verbose output
  --help         Show this message and exit.

Commands:
  clear    Clear all documents in logs and metrics data streams
  destroy  Stop and remove containers, networks, and volumes
  help     Show this help message
  restart  Restart all Elastic Stack containers
  stage    Download all necessary Docker images to local storage
  start    Create container network and start all stack containers
  status   Check the status of all stack containers
  stop     Stop running containers without removing them

Enable Pre-Built Detection Rules

If you want to bulk enable Elastic's pre-built detection rules by OS, on startup, you can change the value of the chosen OS in the .env file from 0 to 1.

# Bulk Enable Detection Rules by OS
LinuxDR=0

WindowsDR=1

MacOSDR=0

Starting

Starting will:

create a network called elastic
download the Elasticsearch, Kibana, and Elastic-Agent Docker images defined in the script
start Elasticsearch, Kibana, and the Elastic-Agent configured as a Fleet Server w/all settings needed for Fleet and the Detection Engine

Activate virtual environment (if not already active)  
$ source venv/bin/activate

Start the stack  
$ python elastic-container.py start

...
 ⠿ Container elasticsearch-security-setup  Healthy 7.3s  
 ⠿ Container elasticsearch                 Healthy 39.3s  
 ⠿ Container kibana                        Healthy 59.3s  
 ⠿ Container elastic-agent                 Started 59.7s  

Attempting to enable the Detection Engine and Prebuilt-Detection Rules

Kibana is up. Proceeding

Detection engine enabled. Installing prepackaged rules.

Prepackaged rules installed!

Waiting 40 seconds for Fleet Server setup

Populating Fleet Settings

Browse to https://localhost:5601

After a few minutes, when prompted, browse to https://localhost:5601 and log in with your configured credentials.

Subsequent starts are much faster - if containers already exist and are running, the script will detect this and complete instantly. If containers exist but are stopped, it will start them quickly (~30 seconds) without reconfiguring.

Destroying

Destroying will:

stop the Elasticsearch and Kibana containers
delete the Elasticsearch and Kibana containers
delete the elastic container network
delete the created volumes

$ python elastic-container.py destroy

fleet-server
kibana
elasticsearch
elastic

Stopping

Stopping will:

stop the Elasticsearch and Kibana containers without deleting them

$ python elastic-container.py stop

fleet-server
kibana
elasticsearch
elastic

Restarting

Restarting will:

restart all the containers

$ python elastic-container.py restart

elasticsearch
kibana
fleet-server

Status

Requesting the status will:

return the current status of the running containers

$ python elastic-container.py status

NAMES: STATUS
fleet-server: Up 6 minutes
kibana: Up 6 minutes
elasticsearch: Up 6 minutes

Clearing

Clearing will :

clear all documents in logs and metrics indices

$ python elastic-container.py clear

Successfully cleared logs data stream
Successfully cleared metrics data stream

Staging

Staging the container images will:

download all container images to your local system, but will not start them

$ python elastic-container.py stage

8.6.0: Pulling from elasticsearch/elasticsearch
e7bd69ff4774: Pull complete
d0a0f12aaf30: Pull complete
...

Verbose Output

All commands support a verbose flag (-v) for detailed output:

$ python elastic-container.py -v start

Force Reconfiguration

If you need to reconfigure Detection Engine or Fleet settings after they've already been set up:

$ python elastic-container.py --force-setup start

Modifying

In .env, the variables are defined, below are the variables that can be changed. You must change the default passwords.

ELASTIC_PASSWORD="changeme"
KIBANA_PASSWORD="changeme"
STACK_VERSION="9.2.1"

If you want to change the default values, simply replace whatever is appropriate in the variable declaration.

If you want to use different Elastic Stack versions, you can change those as well. Optional values are on Elastic's Docker hub:

Increase JVM Heap Size

The default heap size is 512M which may be insufficent in some cases. In that case we can change the heap size by editing docker-compose.yml and passing ES_JAVA_OPTS environment variable to elasticsearch container.

  elasticsearch:
   ...
    environment:
+     - ES_JAVA_OPTS=-Xmx1g -Xms1g

Automating

To enroll an Agent you will need the enrollment token. You can get the token either under https://<KIBANAHOST>:5601/app/fleet/enrollment-tokens or via the API https://www.elastic.co/guide/en/fleet/current/fleet-api-docs.html#get-enrollment-token-api

curl -k --request GET \
   --url 'https://<KIBANAHOST>:5601/api/fleet/enrollment_api_keys' \
   -u <USER>:<PASSWORD> \
   --header 'Content-Type: application/json' \
   --header 'kbn-xsrf: xx'

This will return the tokens in JSON:

{
  "list": [
    {
      "id": "461cc77f-e9dd-46f0-b5c8-7babf644b08f",
      "active": true,
      "api_key_id": "ZS7TYI4B02xLEiUBWuqK",
      "api_key": "WlM3VFlJNEIwMnhMRWlVQld1cUs6b3JmRGRyTnBUSmVOc05DeU1NelJIZw==",
      "name": "Default (461cc77f-e9dd-46f0-b5c8-7babf644b08f)",
      "policy_id": "09528aeb-70c7-4448-91cf-0be1e6a1838a",
      "created_at": "2024-03-21T11:44:08.721Z"
    },
[...]

With that information it is possible to enroll an Agent, e.g. via WinRM or Ansible:

$ProgressPreference = 'SilentlyContinue'
Invoke-WebRequest -Uri https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-[version]-windows-x86_64.zip -OutFile elastic-agent-[version]-windows-x86_64.zip
Expand-Archive .\elastic-agent-[version]-windows-x86_64.zip -DestinationPath .
cd elastic-agent-[version]-windows-x86_64
.\elastic-agent.exe install --url=https://<FLEETHOST>:8220 --insecure -f --enrollment-token=<api_key>