admin/trailofbits-buttercup
1
0
Fork 0
mirror of https://github.com/trailofbits/buttercup synced 2026-06-21 14:11:39 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(scripts): add docker-only e2e command for example-libpng

Browse Source 
Adds scripts/e2e.sh, `make e2e`, and a .claude/commands/e2e.md slash
command that bring the Buttercup stack up via dev/docker-compose
(no Kubernetes), submit the example-libpng task, and monitor the
scheduler / seed-gen / patcher logs through the milestones tracked by
.github/workflows/system-integration.yml (fuzzer build, POV submit/
pass, seed-gen, patch generate / approve / pass, bundle submit, and
optionally SARIF). Defaults LITELLM_MAX_BUDGET to \$3 so accidental
runs are cheap; tears the stack down on exit unless --keep-up is set.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Riccardo Schirone
2026-05-14 16:31:48 +02:00
parent 4cc95c6659
commit b370394146
3 changed files with 520 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.claude/commands/e2e.md
+89
View File
@@ -0,0 +1,89 @@
---
description: Run a Docker-only end-to-end smoke test of Buttercup against example-libpng with a low LLM budget, and monitor the pipeline.
argument-hint: "[--budget N] [--task-duration SEC] [--keep-up] [--no-build] [--skip-wait] [--sarif]"
allowed-tools: Bash(./scripts/e2e.sh:*), Bash(make e2e*), Bash(docker compose:*), Bash(cd dev/docker-compose && docker compose:*), Read
---
# /e2e — Docker-only end-to-end Buttercup run (example-libpng)
This command exercises the full Buttercup pipeline on the [example-libpng](https://github.com/tob-challenges/example-libpng) challenge **using Docker only — no Kubernetes/minikube**. It uses the `dev/docker-compose/` stack and a low LiteLLM budget (default **$3**), so an accidental run is cheap.
> **Host requirement:** x86_64. The fuzzer / patcher / seed-gen images build on `gcr.io/oss-fuzz-base/base-runner`, which is amd64-only. On aarch64 the build will fail with `exec format error` unless you install `qemu-user-static` + `binfmt` and set `DOCKER_DEFAULT_PLATFORM=linux/amd64` (and even then everything runs ~10× slower under emulation).
Mirrors the milestones in `.github/workflows/system-integration.yml`, but tails `docker compose logs` instead of `kubectl logs`.
## What it does
1. Checks for `docker`, `docker compose`, `curl`, and at least one LLM provider key (`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, or `GEMINI_API_KEY`) in your env.
2. Writes `dev/docker-compose/.env` with the provider keys and `LITELLM_MAX_BUDGET=$BUDGET` (default `3`).
3. Builds and starts every service in `dev/docker-compose/compose.yaml` (redis, dind, litellm, task-server, task-downloader, scheduler, program-model, build-bot, fuzzer-bot, coverage-bot, tracer-bot, seed-gen, patcher, buttercup-ui).
4. POSTs the canned libpng `trigger_task` payload to `http://localhost:31323/webhook/trigger_task`.
5. Waits, in order, for these scheduler/seed-gen log markers (timeout configurable per phase):
- `Processing build output for type FUZZER` — fuzzer build done
- `POV submission response: pov_id=` — vulnerability found and POV submitted
- `Updated POV status. New status PASSED` — POV accepted by competition API
- `Copied N files to corpus` — seed-gen produced seeds
- `Appending patch for task` — patch generated
- approves the patch via `POST /v1/task/<task_id>/patch/<patch_id>/approve`
- `Patch passed` — patch accepted
- `Bundle submission response: bundle_id=` — bundle submitted
6. With `--sarif`, also sends a SARIF broadcast and waits for `Matching SARIF submission response`.
7. Prints a colored summary and tears the stack down with `docker compose down -v` (unless `--keep-up`).
## Run it
The driver is `scripts/e2e.sh`. The `Makefile` exposes `make e2e`.
```bash
# Plain run with the $3 budget default
make e2e
# Pass flags through the Makefile
make e2e E2E_ARGS="--budget 5 --keep-up"
# Or call the script directly
./scripts/e2e.sh --budget 3 --task-duration 1800
./scripts/e2e.sh --skip-wait --keep-up # just bring the stack up + submit task
./scripts/e2e.sh --sarif # also exercise the SARIF flow
```
The script writes/overwrites `dev/docker-compose/.env` on each run.
## Monitoring while it's running
The script already streams milestone progress to its own stdout. For finer-grained visibility while it runs:
```bash
# All services, follow
cd dev/docker-compose && docker compose logs -f
# Just the scheduler (most milestones live here)
cd dev/docker-compose && docker compose logs -f scheduler
# Patcher, seed-gen, fuzzer-bot, program-model
cd dev/docker-compose && docker compose logs -f patcher seed-gen fuzzer-bot program-model
# LiteLLM spend tracking
cd dev/docker-compose && docker compose logs -f litellm | grep -i 'spend\|budget'
```
The web UI is at `http://localhost:31323` (no port-forward needed — it's published on the host).
## Tearing down
```bash
cd dev/docker-compose && docker compose down -v --remove-orphans
```
`scripts/e2e.sh` does this automatically on exit unless you pass `--keep-up`.
## When you invoke /e2e
When the user runs `/e2e`, default behavior:
1. Run `./scripts/e2e.sh $ARGUMENTS` (forwarding any flags the user passed).
2. While it runs, surface key transitions to the user. The script's own output already prints `[e2e] Reached: …` for each milestone — relay those as they arrive.
3. If the run fails on a milestone, fetch the last ~50 lines of the relevant service:
- `cd dev/docker-compose && docker compose logs --tail=50 <service>`
4. If the user asks to keep digging, expand the watch with `docker compose logs -f <service>` until the user is satisfied.
5. On success, summarize the milestones reached and remind the user the stack is already torn down (or still up, if `--keep-up`).
Makefile
+7 -1
View File
@@ -1,6 +1,6 @@
# Makefile for Trail of Bits AIxCC Finals CRS
.PHONY: help setup-local setup-azure validate deploy test undeploy install-cscope lint lint-component clean-local wait-crs check-crs crs-instance-id status send-integration-task
.PHONY: help setup-local setup-azure validate deploy test undeploy install-cscope lint lint-component clean-local wait-crs check-crs crs-instance-id status send-integration-task e2e
# Default target
help:
@@ -23,6 +23,7 @@ help:
@echo "Testing:"
@echo " send-integration-task - Run integration-test task"
@echo " send-libpng-task - Run libpng task"
@echo " e2e - Docker-only end-to-end smoke test against example-libpng (low LLM budget)"
@echo ""
@echo "Development:"
@echo " install-cscope - Install cscope tool"
@@ -150,6 +151,11 @@ send-libpng-task:
./orchestrator/scripts/task_crs.sh; \
kill $$PORT_FORWARD_PID 2>/dev/null || true
# Docker-only end-to-end run against example-libpng. No Kubernetes required.
# Pass extra flags via E2E_ARGS, e.g.: make e2e E2E_ARGS="--keep-up --budget 5"
e2e:
@./scripts/e2e.sh $(E2E_ARGS)
# Development targets
lint:
@echo "Linting all Python code..."
scripts/e2e.sh
Executable
+424
View File
@@ -0,0 +1,424 @@
#!/usr/bin/env bash
# scripts/e2e.sh — Run the full Buttercup pipeline against example-libpng using
# the dev docker-compose stack (no Kubernetes required).
#
# This mirrors the milestones checked by .github/workflows/system-integration.yml
# but reads docker-compose logs instead of `kubectl logs`.
set -u
set -o pipefail
###############################################################################
# Config & defaults
###############################################################################
# Resolve repo root from this script's location.
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
COMPOSE_DIR="${REPO_ROOT}/dev/docker-compose"
ENV_FILE="${COMPOSE_DIR}/.env"
# Defaults — overridable via flags or environment.
BUDGET="${LITELLM_MAX_BUDGET:-3}"
TASK_DURATION="${E2E_TASK_DURATION:-1800}"
BUILD_TIMEOUT="${E2E_BUILD_TIMEOUT:-1800}" # seconds (fuzzer build)
VULN_TIMEOUT="${E2E_VULN_TIMEOUT:-1800}"
PATCH_TIMEOUT="${E2E_PATCH_TIMEOUT:-1800}"
BUNDLE_TIMEOUT="${E2E_BUNDLE_TIMEOUT:-300}"
SEED_GEN_TIMEOUT="${E2E_SEED_GEN_TIMEOUT:-1800}"
DO_BUILD=1
DO_TEARDOWN=1
SKIP_WAIT=0
TASK_JSON="" # if set, used instead of the canned libpng payload
SARIF_RUN=0
###############################################################################
# Logging
###############################################################################
if [[ -t 1 ]]; then
C_RST=$'\033[0m'; C_RED=$'\033[1;31m'; C_GRN=$'\033[1;32m'
C_YLW=$'\033[1;33m'; C_BLU=$'\033[1;36m'; C_DIM=$'\033[2m'
else
C_RST=""; C_RED=""; C_GRN=""; C_YLW=""; C_BLU=""; C_DIM=""
fi
log() { printf '%s[e2e]%s %s\n' "$C_BLU" "$C_RST" "$*"; }
ok() { printf '%s[e2e]%s %s\n' "$C_GRN" "$C_RST" "$*"; }
warn() { printf '%s[e2e]%s %s\n' "$C_YLW" "$C_RST" "$*" >&2; }
err() { printf '%s[e2e]%s %s\n' "$C_RED" "$C_RST" "$*" >&2; }
dim() { printf '%s[e2e]%s %s%s%s\n' "$C_BLU" "$C_RST" "$C_DIM" "$*" "$C_RST"; }
###############################################################################
# Usage
###############################################################################
usage() {
cat <<EOF
Usage: scripts/e2e.sh [options]
Runs an end-to-end smoke test of Buttercup against example-libpng using
docker-compose (no Kubernetes). Monitors scheduler/seed-gen logs for the
milestones tracked by .github/workflows/system-integration.yml.
Options:
--budget DOLLARS LiteLLM per-user max budget (default: $BUDGET)
--task-duration SECONDS How long the CRS should fuzz (default: $TASK_DURATION)
--task-json FILE Custom trigger_task payload (default: example-libpng)
--no-build Skip 'docker compose build' (use existing images)
--keep-up Don't tear the stack down on exit (for debugging)
--skip-wait Bring the stack up and submit the task, but don't
block waiting on milestones (returns immediately)
--sarif Also submit a SARIF broadcast after the patch
passes and wait for the matching SARIF response
--build-timeout SEC Override fuzzer-build milestone timeout (default $BUILD_TIMEOUT)
--vuln-timeout SEC Override vuln milestone timeout (default $VULN_TIMEOUT)
--patch-timeout SEC Override patch milestone timeout (default $PATCH_TIMEOUT)
--bundle-timeout SEC Override bundle milestone timeout (default $BUNDLE_TIMEOUT)
--seed-gen-timeout SEC Override seed-gen milestone timeout (default $SEED_GEN_TIMEOUT)
-h, --help Print this help
Required environment (at least one provider key, plus litellm master key):
ANTHROPIC_API_KEY and/or OPENAI_API_KEY and/or GEMINI_API_KEY
BUTTERCUP_LITELLM_KEY (optional, defaults to sk-1234 for local runs)
Optional:
LANGFUSE_HOST, LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY
The script writes ${ENV_FILE} from the values above each run.
EOF
}
###############################################################################
# Argument parsing
###############################################################################
while [[ $# -gt 0 ]]; do
case "$1" in
--budget) BUDGET="$2"; shift 2 ;;
--task-duration) TASK_DURATION="$2"; shift 2 ;;
--task-json) TASK_JSON="$(cat "$2")"; shift 2 ;;
--no-build) DO_BUILD=0; shift ;;
--keep-up) DO_TEARDOWN=0; shift ;;
--skip-wait) SKIP_WAIT=1; shift ;;
--sarif) SARIF_RUN=1; shift ;;
--build-timeout) BUILD_TIMEOUT="$2"; shift 2 ;;
--vuln-timeout) VULN_TIMEOUT="$2"; shift 2 ;;
--patch-timeout) PATCH_TIMEOUT="$2"; shift 2 ;;
--bundle-timeout) BUNDLE_TIMEOUT="$2"; shift 2 ;;
--seed-gen-timeout) SEED_GEN_TIMEOUT="$2"; shift 2 ;;
-h|--help) usage; exit 0 ;;
*) err "Unknown argument: $1"; usage; exit 2 ;;
esac
done
###############################################################################
# Pre-flight checks
###############################################################################
if ! command -v docker >/dev/null 2>&1; then
err "docker is required but not installed."
exit 1
fi
if ! docker compose version >/dev/null 2>&1; then
err "'docker compose' v2 is required (not 'docker-compose')."
exit 1
fi
if ! command -v curl >/dev/null 2>&1; then
err "curl is required but not installed."
exit 1
fi
provider_keys_set=0
for v in ANTHROPIC_API_KEY OPENAI_API_KEY GEMINI_API_KEY; do
val="${!v:-}"
if [[ -n "$val" && "$val" != "<INSERT_KEY>" ]]; then
provider_keys_set=1
fi
done
if [[ "$provider_keys_set" -eq 0 ]]; then
err "No LLM provider key found in env. Set ANTHROPIC_API_KEY, OPENAI_API_KEY, or GEMINI_API_KEY."
err "Tip: 'export ANTHROPIC_API_KEY=...; scripts/e2e.sh' or add to ${ENV_FILE} first."
exit 1
fi
# If keys are missing, leave them at the placeholder so litellm still loads the
# config (some models will fail at request time, others will succeed).
: "${ANTHROPIC_API_KEY:=<INSERT_KEY>}"
: "${OPENAI_API_KEY:=<INSERT_KEY>}"
: "${GEMINI_API_KEY:=<INSERT_KEY>}"
: "${AZURE_API_BASE:=<INSERT_HOST>}"
: "${AZURE_API_KEY:=<INSERT_KEY>}"
: "${BUTTERCUP_LITELLM_KEY:=sk-1234}"
: "${LANGFUSE_HOST:=}"
: "${LANGFUSE_PUBLIC_KEY:=}"
: "${LANGFUSE_SECRET_KEY:=}"
###############################################################################
# .env generation
###############################################################################
log "Writing ${ENV_FILE} (LITELLM_MAX_BUDGET=\$${BUDGET})"
{
echo "# Generated by scripts/e2e.sh on $(date -Is)"
echo "BUTTERCUP_LITELLM_KEY=${BUTTERCUP_LITELLM_KEY}"
echo "ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}"
echo "OPENAI_API_KEY=${OPENAI_API_KEY}"
echo "GEMINI_API_KEY=${GEMINI_API_KEY}"
echo "AZURE_API_BASE=${AZURE_API_BASE}"
echo "AZURE_API_KEY=${AZURE_API_KEY}"
echo "LITELLM_MAX_BUDGET=${BUDGET}"
echo "LANGFUSE_HOST=${LANGFUSE_HOST}"
echo "LANGFUSE_PUBLIC_KEY=${LANGFUSE_PUBLIC_KEY}"
echo "LANGFUSE_SECRET_KEY=${LANGFUSE_SECRET_KEY}"
} > "$ENV_FILE"
###############################################################################
# docker compose helpers
###############################################################################
# Always run compose from the compose dir so relative includes resolve.
dc() {
(cd "$COMPOSE_DIR" && docker compose "$@")
}
teardown() {
if [[ "$DO_TEARDOWN" -eq 1 ]]; then
log "Tearing the stack down (docker compose down -v)"
dc down -v --remove-orphans || true
else
warn "Leaving the stack up (--keep-up). Tear down with: cd ${COMPOSE_DIR} && docker compose down -v"
fi
}
on_exit() {
rc=$?
teardown
if [[ $rc -ne 0 ]]; then
err "e2e run finished with exit code $rc"
fi
exit $rc
}
trap on_exit EXIT INT TERM
###############################################################################
# Bring the stack up
###############################################################################
if [[ "$DO_BUILD" -eq 1 ]]; then
log "Building docker compose images (this can take a while the first time)"
if ! dc build; then
err "docker compose build failed. On non-x86_64 hosts this usually means an"
err "image (e.g. fuzzer/Dockerfile -> gcr.io/oss-fuzz-base/base-runner) is amd64-only."
err "Inspect the build output above; retry on an x86_64 host, or install"
err "qemu-user-static + binfmt and re-run with DOCKER_DEFAULT_PLATFORM=linux/amd64."
exit 1
fi
fi
log "Starting services"
if ! dc up -d; then
err "docker compose up failed. Check 'docker compose ps' / logs."
exit 1
fi
# Wait for the buttercup-ui task webhook to be reachable.
log "Waiting for buttercup-ui to accept connections on http://localhost:31323"
ui_up=0
for _ in $(seq 1 120); do
if curl -sf -o /dev/null -m 2 http://localhost:31323/v1/ping/ 2>/dev/null \
|| curl -sf -o /dev/null -m 2 http://localhost:31323/ 2>/dev/null; then
ui_up=1; break
fi
sleep 2
done
if [[ "$ui_up" -ne 1 ]]; then
err "buttercup-ui did not come up on port 31323. Check 'docker compose logs buttercup-ui'."
exit 1
fi
ok "buttercup-ui is up."
###############################################################################
# Submit the task
###############################################################################
if [[ -z "$TASK_JSON" ]]; then
TASK_JSON=$(cat <<EOF
{
"challenge_repo_url": "https://github.com/tob-challenges/example-libpng",
"challenge_repo_base_ref": "5bf8da2d7953974e5dfbd778429c3affd461f51a",
"challenge_repo_head_ref": "challenges/lp-delta-01",
"fuzz_tooling_url": "https://github.com/trail-of-forks/oss-fuzz",
"fuzz_tooling_ref": "fix-libpng",
"fuzz_tooling_project_name": "libpng",
"duration": ${TASK_DURATION}
}
EOF
)
fi
log "Submitting task to buttercup-ui /webhook/trigger_task"
http_code=$(curl -s -o /tmp/e2e_task_resp.$$ -w '%{http_code}' \
-X POST 'http://127.0.0.1:31323/webhook/trigger_task' \
-H 'Content-Type: application/json' \
-d "$TASK_JSON")
resp_body=$(cat /tmp/e2e_task_resp.$$ || true)
rm -f /tmp/e2e_task_resp.$$
if [[ "$http_code" != "200" && "$http_code" != "201" ]]; then
err "trigger_task returned HTTP $http_code: $resp_body"
exit 1
fi
ok "Task accepted (HTTP $http_code). ${C_DIM}${resp_body}${C_RST}"
if [[ "$SKIP_WAIT" -eq 1 ]]; then
ok "--skip-wait set; not waiting on milestones."
DO_TEARDOWN=0
exit 0
fi
###############################################################################
# Milestone waiters
###############################################################################
# wait_for SERVICE PATTERN TIMEOUT_SEC LABEL
#
# Tails `docker compose logs <SERVICE>` until a line matching PATTERN appears
# or TIMEOUT_SEC elapses. Returns 0 on success, non-zero on timeout.
wait_for() {
local service="$1" pattern="$2" timeout="$3" label="$4"
local deadline=$(( $(date +%s) + timeout ))
log "Waiting for milestone: ${label} ${C_DIM}(service=${service}, timeout=${timeout}s)${C_RST}"
while [[ $(date +%s) -lt $deadline ]]; do
# --no-color so the grep matches plain text; --tail=all replays history.
if dc logs --no-color --no-log-prefix --tail=all "$service" 2>/dev/null \
| grep -m1 -E "$pattern" >/dev/null; then
ok "Reached: ${label}"
return 0
fi
sleep 15
done
err "Timed out after ${timeout}s waiting for: ${label}"
err "Recent logs from ${service}:"
dc logs --no-color --tail=50 "$service" >&2 || true
return 1
}
# Capture a single matching log line (returns it on stdout, empty on miss).
capture_line() {
local service="$1" pattern="$2"
dc logs --no-color --no-log-prefix --tail=all "$service" 2>/dev/null \
| grep -E "$pattern" | head -n1 || true
}
###############################################################################
# Walk through the pipeline
###############################################################################
declare -a SUMMARY=()
record() { SUMMARY+=("$1"); }
wait_for scheduler \
"Processing build output for type FUZZER" \
"$BUILD_TIMEOUT" "fuzzer build processed" \
&& record "fuzzer-build: ok" || record "fuzzer-build: TIMEOUT"
wait_for scheduler \
"POV submission response: pov_id=" \
"$VULN_TIMEOUT" "vulnerability (POV) submitted" \
&& record "pov-submit: ok" || record "pov-submit: TIMEOUT"
wait_for scheduler \
"Updated POV status. New status PASSED" \
"$VULN_TIMEOUT" "POV accepted by competition API" \
&& record "pov-passed: ok" || record "pov-passed: TIMEOUT"
wait_for seed-gen \
"Copied [1-9][0-9]* files to corpus" \
"$SEED_GEN_TIMEOUT" "seed-gen produced seeds" \
&& record "seed-gen: ok" || record "seed-gen: TIMEOUT"
wait_for scheduler \
"Appending patch for task" \
"$PATCH_TIMEOUT" "patch generated" \
&& record "patch-generated: ok" || record "patch-generated: TIMEOUT"
# Approve the patch (the local UI requires explicit approval, unlike scored
# rounds where it is automatic).
PATCH_LINE="$(capture_line scheduler 'competition_patch_id=')"
if [[ -n "$PATCH_LINE" ]]; then
PATCH_ID=$(printf '%s' "$PATCH_LINE" | sed -n 's/.*competition_patch_id=\([^ ]*\).*/\1/p')
# Task id is inside the first [...] block, after the last ':'.
TASK_ID=$(printf '%s' "$PATCH_LINE" | sed -n 's/.*\[\([^]]*\)\].*/\1/p' | sed 's/^[^:]*://')
if [[ -n "$PATCH_ID" && -n "$TASK_ID" ]]; then
log "Approving patch ${C_DIM}task=${TASK_ID} patch=${PATCH_ID}${C_RST}"
curl -fsS -X POST \
"http://127.0.0.1:31323/v1/task/${TASK_ID}/patch/${PATCH_ID}/approve" \
>/dev/null && record "patch-approve: ok" || record "patch-approve: HTTP fail"
else
warn "Could not extract patch/task ids from: $PATCH_LINE"
record "patch-approve: skipped (parse fail)"
fi
else
warn "No competition_patch_id= line seen; skipping approval"
record "patch-approve: skipped (no patch line)"
fi
wait_for scheduler \
"Patch passed" \
"$PATCH_TIMEOUT" "patch accepted by competition API" \
&& record "patch-passed: ok" || record "patch-passed: TIMEOUT"
wait_for scheduler \
"Bundle submission response: bundle_id=" \
"$BUNDLE_TIMEOUT" "bundle submitted" \
&& record "bundle-submit: ok" || record "bundle-submit: TIMEOUT"
if [[ "$SARIF_RUN" -eq 1 ]]; then
SARIF_TASK_ID="${TASK_ID:-}"
if [[ -z "$SARIF_TASK_ID" ]]; then
SARIF_TASK_ID=$(dc logs --no-color --no-log-prefix --tail=all scheduler \
| grep "Submitting bundle for harness" | head -n1 \
| grep -o "\[[^]]*\]" | head -n1 \
| tr -d '[]' | awk -F: '{print $NF}')
fi
if [[ -n "$SARIF_TASK_ID" ]]; then
log "Sending SARIF broadcast for task ${SARIF_TASK_ID}"
if "${REPO_ROOT}/orchestrator/scripts/send_sarif.sh" "$SARIF_TASK_ID" >/dev/null 2>&1; then
record "sarif-send: ok"
else
record "sarif-send: HTTP fail"
fi
wait_for scheduler \
"Matching SARIF submission response" \
"$BUNDLE_TIMEOUT" "SARIF accepted" \
&& record "sarif-passed: ok" || record "sarif-passed: TIMEOUT"
else
record "sarif: skipped (no task id)"
fi
fi
###############################################################################
# Summary
###############################################################################
printf '\n%s===================== e2e summary =====================%s\n' "$C_BLU" "$C_RST"
for line in "${SUMMARY[@]}"; do
if [[ "$line" == *": ok" ]]; then
printf ' %s✓%s %s\n' "$C_GRN" "$C_RST" "$line"
elif [[ "$line" == *": TIMEOUT" || "$line" == *"fail"* ]]; then
printf ' %s✗%s %s\n' "$C_RED" "$C_RST" "$line"
else
printf ' %s•%s %s\n' "$C_YLW" "$C_RST" "$line"
fi
done
printf '%s=======================================================%s\n' "$C_BLU" "$C_RST"
# Exit non-zero if any milestone failed.
for line in "${SUMMARY[@]}"; do
if [[ "$line" == *": TIMEOUT" || "$line" == *"fail"* ]]; then
exit 1
fi
done