# The Elastic Container Project Stand up a 100% containerized Elastic stack, TLS secured, with Elasticsearch, Kibana, Fleet, and the Detection Engine all pre-configured, enabled and ready to use, within minutes. If you're interested in more details regarding this project and what to do once you have it running, check out our [blog post](https://www.elastic.co/security-labs/the-elastic-container-project) on the Elastic Security Labs site. :warning: This is not an Elastic created, sponsored, or maintained project. Elastic is not responsible for this projects design or implementation.

ecp-logo

## Steps 1. `Git clone` this repo 2. Install prerequisites (see below) 3. Change into the `elastic-container/` folder 4. Change the default password of `changeme` in the `.env` file (don't change the `elastic` username, it's a [required built-in user](https://www.elastic.co/guide/en/elasticsearch/reference/current/built-in-users.html)) 5. Bulk enable pre-built detection rules by OS in the `.env` file (not required, see usage below) 6. Create a Python virtual environment: `python -m venv venv` 7. Activate the virtual environment and install dependencies: `source venv/bin/activate && pip install -r requirements.txt` 8. Execute the Python script with the start command: `python elastic-container.py start` 9. Wait for the prompt to tell you to browse to https://localhost:5601 ## Requirements ### Operating System: - Linux (Ubuntu, Debian, Rocky, Fedora, RHEL, CentOS) - MacOS - Windows (via WSL2) ### Prerequisites: - **Python 3.10 or higher** (required) - [Docker suite](https://docs.docker.com/get-docker/) with [docker compose plugin](https://docs.docker.com/compose/install/) - [Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) **MacOS:** ```bash # Install Homebrew if not already installed /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" # Install prerequisites brew install python@3.11 git brew install --cask docker ``` Once we have Docker installed we need to provide it with privileged access for it to function. Run the following command to open the Docker app and follow the proceeding steps. ```bash open /Applications/Docker.app ``` 1. Confirm you would like to open the app 2. Select ok when prompted to provide Docker with privileged access 3. Enter your password 4. Close or minimize the Docker app **Ubuntu/Debian:** Please follow the [Docker installation instructions](https://docs.docker.com/engine/install/ubuntu/). Of specific note, you *must* install the `docker-compose-plugin`, which is different than `docker-compose`. ```bash # Install software-properties-common for PPA support sudo apt-get update sudo apt-get install -y software-properties-common # Add Python 3.11 repository (deadsnakes PPA) sudo add-apt-repository ppa:deadsnakes/ppa -y sudo apt-get update # Install Python 3.11 and tools sudo apt-get install -y python3.11 python3.11-venv python3.11-distutils python3-pip git # Install Docker curl -fsSL https://get.docker.com -o get-docker.sh sudo sh get-docker.sh sudo apt-get install -y docker-compose-plugin # Add user to docker group and start Docker sudo usermod -aG docker $USER newgrp docker sudo systemctl start docker sudo systemctl enable docker ``` **RPM distributions (Rocky/RHEL/CentOS/Fedora):** Please follow the [Docker installation instructions](https://docs.docker.com/engine/install/centos/). Of specific note, you *must* install the `docker-compose-plugin`, which is different than `docker-compose`. ```bash # Install Python and tools sudo dnf install -y python3.11 python3-pip git dnf-plugins-core # Add Docker's official repository sudo dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo # Install Docker Engine and Compose plugin sudo dnf install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin # Add user to docker group and start Docker sudo usermod -aG docker $USER newgrp docker sudo systemctl start docker sudo systemctl enable docker ``` **Windows 10/11 with WSL 2 (Ubuntu 20.04):** Make sure you are using WSL version 2. You can check the version using `wsl -l -v` in PowerShell. If the version is wrong you can change it with `wsl --set-version Ubuntu-20.04 2` ```bash sudo apt-get update # Follow Ubuntu installation instructions above sudo apt-get install -y software-properties-common sudo add-apt-repository ppa:deadsnakes/ppa -y sudo apt-get update sudo apt-get install -y python3.11 python3.11-venv python3.11-distutils python3-pip git ``` Please follow the [Docker installation instructions](https://docs.docker.com/engine/install/ubuntu/). Of specific note, you *must* install the `docker-compose-plugin`, which is different than `docker-compose`. Once the Docker suite is installed run `sudo service docker start` to start it. ## Usage This uses default creds of `elastic:changeme` and is intended purely for security research on a local Elastic stack. [Change the password in the `.env` file](https://github.com/peasead/elastic-container/blob/main/README.md#modifying). Don't change the `elastic` username, it's a [required built-in user](https://www.elastic.co/guide/en/elasticsearch/reference/current/built-in-users.html) This should not be Internet exposed or used in a production environment. ```bash Usage: elastic-container.py [OPTIONS] COMMAND [ARGS]... Elastic Container - Manage your containerized Elastic Stack Options: -v, --verbose Enable verbose output --help Show this message and exit. Commands: clear Clear all documents in logs and metrics data streams destroy Stop and remove containers, networks, and volumes help Show this help message restart Restart all Elastic Stack containers stage Download all necessary Docker images to local storage start Create container network and start all stack containers status Check the status of all stack containers stop Stop running containers without removing them ``` ### Enable Pre-Built Detection Rules If you want to bulk enable Elastic's pre-built detection rules by OS, on startup, you can change the value of the chosen OS in the `.env` file from 0 to 1. ``` # Bulk Enable Detection Rules by OS LinuxDR=0 WindowsDR=1 MacOSDR=0 ``` ### Starting Starting will: - create a network called `elastic` - download the Elasticsearch, Kibana, and Elastic-Agent Docker images defined in the script - start Elasticsearch, Kibana, and the Elastic-Agent configured as a Fleet Server w/all settings needed for Fleet and the Detection Engine ``` Activate virtual environment (if not already active) $ source venv/bin/activate Start the stack $ python elastic-container.py start ... ⠿ Container elasticsearch-security-setup Healthy 7.3s ⠿ Container elasticsearch Healthy 39.3s ⠿ Container kibana Healthy 59.3s ⠿ Container elastic-agent Started 59.7s Attempting to enable the Detection Engine and Prebuilt-Detection Rules Kibana is up. Proceeding Detection engine enabled. Installing prepackaged rules. Prepackaged rules installed! Waiting 40 seconds for Fleet Server setup Populating Fleet Settings Browse to https://localhost:5601 ``` After a few minutes, when prompted, browse to https://localhost:5601 and log in with your configured credentials. **Subsequent starts are much faster** - if containers already exist and are running, the script will detect this and complete instantly. If containers exist but are stopped, it will start them quickly (~30 seconds) without reconfiguring. ### Destroying Destroying will: - stop the Elasticsearch and Kibana containers - delete the Elasticsearch and Kibana containers - delete the `elastic` container network - delete the created volumes ``` $ python elastic-container.py destroy fleet-server kibana elasticsearch elastic ``` ### Stopping Stopping will: - stop the Elasticsearch and Kibana containers without deleting them ``` $ python elastic-container.py stop fleet-server kibana elasticsearch elastic ``` ### Restarting Restarting will: - restart all the containers ``` $ python elastic-container.py restart elasticsearch kibana fleet-server ``` ### Status Requesting the status will: - return the current status of the running containers ``` $ python elastic-container.py status NAMES: STATUS fleet-server: Up 6 minutes kibana: Up 6 minutes elasticsearch: Up 6 minutes ``` ### Clearing Clearing will : - clear all documents in logs and metrics indices ``` $ python elastic-container.py clear Successfully cleared logs data stream Successfully cleared metrics data stream ``` ### Staging Staging the container images will: - download all container images to your local system, but will not start them ``` $ python elastic-container.py stage 8.6.0: Pulling from elasticsearch/elasticsearch e7bd69ff4774: Pull complete d0a0f12aaf30: Pull complete ... ``` ### Verbose Output All commands support a verbose flag (`-v`) for detailed output: ``` $ python elastic-container.py -v start ``` ### Force Reconfiguration If you need to reconfigure Detection Engine or Fleet settings after they've already been set up: ``` $ python elastic-container.py --force-setup start ``` ## Modifying In `.env`, the variables are defined, below are the variables that can be changed. **You must change the default passwords.** ``` ELASTIC_PASSWORD="changeme" KIBANA_PASSWORD="changeme" STACK_VERSION="9.2.1" ``` If you want to change the default values, simply replace whatever is appropriate in the variable declaration. If you want to use different Elastic Stack versions, you can change those as well. Optional values are on Elastic's Docker hub: - [Elasticsearch](https://hub.docker.com/r/elastic/elasticsearch/tags?page=1&ordering=last_updated) - [Kibana](https://hub.docker.com/r/elastic/kibana/tags?page=1&ordering=last_updated) - [Elastic-Agent](https://hub.docker.com/r/elastic/elastic-agent/tags?page=1&ordering=last_updated) ### Increase JVM Heap Size The default heap size is 512M which may be insufficent in some cases. In that case we can change the heap size by editing `docker-compose.yml` and passing `ES_JAVA_OPTS` environment variable to elasticsearch container. ```yml elasticsearch: ... environment: + - ES_JAVA_OPTS=-Xmx1g -Xms1g ``` ## Automating To enroll an Agent you will need the enrollment token. You can get the token either under `https://:5601/app/fleet/enrollment-tokens` or via the API [https://www.elastic.co/guide/en/fleet/current/fleet-api-docs.html#get-enrollment-token-api](https://www.elastic.co/guide/en/fleet/current/fleet-api-docs.html#get-enrollment-token-api) ```bash curl -k --request GET \ --url 'https://:5601/api/fleet/enrollment_api_keys' \ -u : \ --header 'Content-Type: application/json' \ --header 'kbn-xsrf: xx' ``` This will return the tokens in JSON: ```json { "list": [ { "id": "461cc77f-e9dd-46f0-b5c8-7babf644b08f", "active": true, "api_key_id": "ZS7TYI4B02xLEiUBWuqK", "api_key": "WlM3VFlJNEIwMnhMRWlVQld1cUs6b3JmRGRyTnBUSmVOc05DeU1NelJIZw==", "name": "Default (461cc77f-e9dd-46f0-b5c8-7babf644b08f)", "policy_id": "09528aeb-70c7-4448-91cf-0be1e6a1838a", "created_at": "2024-03-21T11:44:08.721Z" }, [...] ``` With that information it is possible to enroll an Agent, e.g. via WinRM or Ansible: ```powershell $ProgressPreference = 'SilentlyContinue' Invoke-WebRequest -Uri https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-[version]-windows-x86_64.zip -OutFile elastic-agent-[version]-windows-x86_64.zip Expand-Archive .\elastic-agent-[version]-windows-x86_64.zip -DestinationPath . cd elastic-agent-[version]-windows-x86_64 .\elastic-agent.exe install --url=https://:8220 --insecure -f --enrollment-token= ```