Add README for running custom challenges (#244)

* add readme for running custom challenges * add custom challenge guide to README --------- Co-authored-by: Michael D Brown <michael.brown@trailofbits.com>
2026-06-21 14:11:39 +00:00 · 2025-08-05 09:09:56 -05:00
parent cbafd1e10a
commit cfd0441584
2 changed files with 377 additions and 0 deletions
@@ -0,0 +1,376 @@
+# Writing Custom Challenges
+
+Buttercup is an automated cyber reasoning system originally developed for DARPA's [AI Cyber Challenge][1]. While it was designed to work seamlessly with existing oss-fuzz projects, you can also configure it to analyze custom projects with some additional setup.
+
+This guide walks you through creating custom challenges for Buttercup, covering everything from the basic input format to debugging common issues.
+
+## Prerequisites
+
+To analyze a custom project, you'll need to provide:
+
+- **Fuzzing harnesses** that serve as entry points for analysis
+- **Build configuration** in a forked oss-fuzz repository
+- **Project metadata** defining how to compile and run your fuzzers
+
+## Challenge Input Format
+
+Buttercup accepts challenges through a JSON configuration that specifies both the target project and the fuzzing infrastructure. Here's a complete example:
+
+```json
+{
+    "challenge_repo_url": "https://github.com/tob-challenges/integration-test.git",
+    "challenge_repo_head_ref": "challenges/integration-test-delta-01",
+    "fuzz_tooling_url": "https://github.com/tob-challenges/oss-fuzz-aixcc",
+    "fuzz_tooling_ref": "challenge-state/integration-test-delta-01",
+    "fuzz_tooling_project_name": "integration-test",
+    "duration": 1800,
+}
+```
+
+### Field Descriptions
+
+**Project Repository Fields:**
+- `challenge_repo_url`: URL to the repository containing your source code
+- `challenge_repo_head_ref`: Git branch, tag, or commit to analyze
+- `challenge_repo_base_ref`: *(Delta mode only)* The "clean" commit before vulnerabilities were introduced
+
+**Fuzzing Infrastructure Fields:**
+- `fuzz_tooling_url`: URL to your forked oss-fuzz repository with build instructions
+- `fuzz_tooling_ref`: Git reference in the oss-fuzz fork that can build your project
+- `fuzz_tooling_project_name`: Directory name under `projects/` in your oss-fuzz fork
+
+**Analysis Configuration:**
+- `duration`: Analysis time limit in seconds (typically 1800-86400)
+
+For additional context, consult the official oss-fuzz documentation on "[Setting Up a new project][2]".
+
+## Writing Fuzzing Harnesses
+
+Fuzzing harnesses are small programs that act as entry points for Buttercup's analysis. They define how to feed random input into your code to discover vulnerabilities.
+
+### Key Principles
+
+- **Coverage is critical**: Buttercup can only find bugs in code paths reached by your harnesses
+- **Keep them simple**: Complex harnesses can introduce their own bugs, which will confuse the analysis
+- **Optimize for speed**: The faster your harness runs, the more test cases Buttercup can explore and the better it can find vulnerabilities
+- **Handle edge cases**: Your harness should gracefully handle empty inputs, malformed data, etc.
+
+### C/C++ Projects (LibFuzzer)
+
+For C/C++ projects, Buttercup looks for harnesses that implement the `LLVMFuzzerTestOneInput` function:
+
+```c
+#include <stdint.h>
+#include <stddef.h>
+
+int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
+    // Your fuzzing logic here
+    // Call the functions you want to test with the provided data
+    return 0;
+}
+```
+
+### Java Projects (Jazzer)
+
+For Java projects, Buttercup looks for harnesses that implement the `fuzzerTestOneInput` method:
+
+```java
+import com.code_intelligence.jazzer.api.FuzzedDataProvider;
+
+public class MyFuzzer {
+    public static void fuzzerTestOneInput(FuzzedDataProvider data) {
+        // Your fuzzing logic here
+        // Use data.consume*() methods to extract data for testing
+    }
+}
+```
+
+### Best Practices
+
+- Keep harnesses simple and focused on one API or functionality
+- Avoid complex setup/teardown that could introduce bugs
+- Handle edge cases (empty input, null pointers, etc.) gracefully
+- Use appropriate data extraction methods for structured input
+- Ensure the harness can handle the full range of possible inputs
+
+See the [libFuzzer Tutorial](https://github.com/google/fuzzing/blob/master/tutorial/libFuzzerTutorial.md) and [Jazzer Documentation](https://github.com/CodeIntelligenceTesting/jazzer) for detailed guidance.
+
+## Setting Up Your OSS-Fuzz Fork
+
+To integrate your project with Buttercup, you'll need to create a fork of the [oss-fuzz](https://github.com/google/oss-fuzz) repository and add build instructions for your specific project.
+
+This setup tells Buttercup how to compile your code and harnesses in a consistent, isolated environment.
+
+### Directory Structure
+
+In your oss-fuzz fork, create a new directory at `projects/your-project-name/` containing these files:
+
+| File | Purpose |
+|------|----------|
+| [`Dockerfile`][11] | Defines the build environment and dependencies |
+| [`build.sh`][12] | Compiles your project and creates fuzzer binaries |
+| [`project.yaml`][13] | Metadata about your project *(recommended)* |
+
+[11]: https://google.github.io/oss-fuzz/getting-started/new-project-guide/#dockerfile
+[12]: https://google.github.io/oss-fuzz/getting-started/new-project-guide/#buildsh
+[13]: https://google.github.io/oss-fuzz/getting-started/new-project-guide/#projectyaml
+
+### Creating the Dockerfile
+
+The Dockerfile sets up your build environment. Start with the oss-fuzz base image and add your project's dependencies:
+
+```dockerfile
+FROM gcr.io/oss-fuzz-base/base-builder
+
+# Install build dependencies
+RUN apt-get update && apt-get install -y \
+    make autoconf automake libtool pkg-config
+
+# Clone your project
+RUN git clone https://github.com/your-org/your-project.git
+WORKDIR /src/your-project
+
+# Copy the build script
+COPY build.sh $SRC/
+```
+
+### Writing the Build Script
+
+The `build.sh` script compiles your project and creates the fuzzer executables. OSS-Fuzz provides environment variables for consistent builds:
+
+```bash
+#!/bin/bash -eu
+
+# Build your project's libraries
+make clean
+make CC="$CC" CXX="$CXX" CFLAGS="$CFLAGS" CXXFLAGS="$CXXFLAGS"
+
+# Compile each fuzzing harness
+# $CC, $CXX, $CFLAGS, $CXXFLAGS are provided by oss-fuzz
+# $LIB_FUZZING_ENGINE links the fuzzing engine
+# $OUT is where fuzzer binaries should be placed
+
+$CC $CFLAGS -I. -c harness.c -o harness.o
+$CXX $CXXFLAGS $LIB_FUZZING_ENGINE harness.o \
+    -L. -lyourlib -o $OUT/harness_fuzzer
+
+# Include seed corpus and dictionaries if available
+if [ -d "seeds" ]; then
+    cp seeds/* $OUT/ 2>/dev/null || true
+fi
+```
+
+### Project Metadata
+
+The `project.yaml` file provides important metadata about your project:
+
+```yaml
+homepage: "https://github.com/your-org/your-project"
+language: c  # or jvm
+primary_contact: "maintainer@example.com"
+
+# Fuzzing configuration
+fuzzing_engines:
+  - libfuzzer
+  
+sanitizers:
+  - address      # detects memory corruption
+  - undefined    # detects undefined behavior
+  - memory       # detects uninitialized reads
+```
+
+## Submitting Your Challenge
+
+With your project repository and oss-fuzz fork ready, you can now submit your challenge to Buttercup for analysis. There are three ways to do this:
+
+### Option 1: Using the Challenge Script
+
+First, add your challenge configuration to the `CHALLENGE_MAP` in `orchestrator/scripts/challenge.py`:
+
+```python
+"my_custom_challenge": {
+    "challenge_repo_url": "https://github.com/your-org/your-project.git",
+    "challenge_repo_head_ref": "main",
+    "fuzz_tooling_url": "https://github.com/your-org/oss-fuzz-fork.git",
+    "fuzz_tooling_ref": "main",
+    "fuzz_tooling_project_name": "your-project",
+    "duration": 3600,
+},
+```
+
+Then run your challenge:
+
+```bash
+cd orchestrator/scripts
+python challenge.py single my_custom_challenge 3600
+```
+
+### Option 2: Direct API Call
+
+For one-off submissions or when testing new configurations, use the HTTP API directly:
+
+```bash
+curl -X 'POST' 'http://localhost:31323/webhook/trigger_task' \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "challenge_repo_url": "https://github.com/your-org/your-project.git",
+    "challenge_repo_head_ref": "main",
+    "fuzz_tooling_url": "https://github.com/your-org/oss-fuzz-fork.git",
+    "fuzz_tooling_ref": "main",
+    "fuzz_tooling_project_name": "your-project",
+    "duration": 3600
+  }'
+```
+
+### Option 3: Web Interface
+
+For a user-friendly approach, run `make web-ui` and open `http://localhost:31323` in your browser. The web interface provides a form where you can enter all the challenge parameters and submit them with a single click.
+
+## Troubleshooting Your Challenge
+
+When a challenge doesn't behave as expected, Buttercup provides several tools to help you diagnose and fix issues. Here's a systematic approach to debugging:
+
+### Step 1: Check System Status
+
+Start by verifying that all Buttercup components are running properly:
+
+```bash
+make status
+```
+
+### Step 2: Examine Logs
+
+Logs are your primary source of diagnostic information:
+
+```bash
+# View recent logs for a specific component
+kubectl logs -n crs -l app=<service-name> --tail=100
+
+# Watch logs in real-time to see what's happening
+kubectl logs -n crs -l app=scheduler --tail=-1 -f
+
+# Collect comprehensive logs from all components
+./deployment/collect-logs.sh
+```
+
+### Step 3: Interactive Debugging
+
+When you need to examine the environment directly:
+
+```bash
+# Access a running container's shell
+kubectl exec -it -n crs <pod-name> -- /bin/bash
+
+# Examine files, run commands, check environment variables
+```
+
+### Step 4: Track Workflow Progress
+
+Monitor how your challenge moves through Buttercup's processing pipeline:
+
+```bash
+# Watch the scheduler's state transitions
+kubectl logs -n crs -l app=scheduler --tail=-1 --prefix | \
+  grep "WAIT_PATCH_PASS -> SUBMIT_BUNDLE"
+
+# This helps identify where processing gets stuck
+```
+
+### Common Issues and Solutions
+
+| Problem | Symptoms | Solution |
+|---------|----------|----------|
+| **Build failures** | Fuzzer compilation errors | Verify your `build.sh` works with oss-fuzz base images. Test locally with `docker run gcr.io/oss-fuzz-base/base-builder` |
+| **Missing harnesses** | "No harnesses found" errors | Ensure harness files contain the expected function signatures |
+| **Timeouts** | Analysis stops prematurely | Increase the `duration` parameter, especially for large codebases |
+| **Resource limits** | Pods getting killed (OOMKilled) | Check pod memory/CPU limits and adjust Kubernetes resource constraints |
+| **Git access issues** | Clone/fetch failures | Verify repository URLs are accessible and authentication is configured |
+| **Path problems** | File not found errors | Ensure WORKDIR in Dockerfile matches your project structure |
+
+## Analysis Modes
+
+Buttercup supports two analysis modes, each optimized for different use cases:
+
+### Full Mode Analysis
+
+Full mode analyzes an entire project:
+
+| Aspect | Details |
+|--------|----------|
+| **Scope** | Analyzes the complete codebase |
+| **Best for** | Initial assessment |
+| **Configuration** | Only requires `challenge_repo_head_ref` |
+
+**Example configuration:**
+
+```json
+{
+    "challenge_repo_url": "https://github.com/example/project.git",
+    "challenge_repo_head_ref": "v1.0.0",
+    "fuzz_tooling_url": "https://github.com/example/oss-fuzz-fork.git",
+    "fuzz_tooling_ref": "main",
+    "fuzz_tooling_project_name": "project",
+    "duration": 86400
+}
+```
+
+### Delta Mode Analysis
+
+Delta mode focuses specifically on changes between two commits, making it ideal for targeted analysis:
+
+| Aspect | Details |
+|--------|----------|
+| **Scope** | Analyzes only the differences between base and head commits |
+| **Best for** | Analyzing specific changes |
+| **Configuration** | Requires both `challenge_repo_base_ref` and `challenge_repo_head_ref` |
+
+**Example configuration:**
+
+```json
+{
+    "challenge_repo_url": "https://github.com/example/project.git",
+    "challenge_repo_base_ref": "abc123",
+    "challenge_repo_head_ref": "def456",
+    "fuzz_tooling_url": "https://github.com/example/oss-fuzz-fork.git",
+    "fuzz_tooling_ref": "main",
+    "fuzz_tooling_project_name": "project",
+    "duration": 28800
+}
+```
+
+### How Mode Detection Works
+
+Buttercup determines which mode to use based on your challenge configuration:
+
+- **Delta mode** is activated when you provide both `challenge_repo_base_ref` and `challenge_repo_head_ref`
+- Internally, Buttercup creates diff files and the `is_delta_mode()` method detects their presence
+- Before building, delta mode applies patches using `git apply` to introduce the target changes
+- This targeted approach makes delta mode more efficient for analyzing specific code changes
+
+### Choosing the Right Mode
+
+**Choose Full Mode when you want to:**
+- Perform an initial assessment of a new project
+- Search for vulnerabilities in all code reachable from harnesses
+- Analyze a project without specific change targets
+
+**Choose Delta Mode when you want to:**
+- Test specific code changes
+- Reproduce known vulnerabilities
+
+---
+
+## Next Steps
+
+Now that you understand how to create custom challenges, you can:
+
+1. **Start simple**: Create a basic challenge with one harness to test your setup
+2. **Test locally**: Verify your oss-fuzz configuration builds correctly before submitting
+3. **Monitor progress**: Use the debugging techniques to track your challenge's execution
+4. **Iterate**: Refine your harnesses and build scripts based on the results
+
+For additional help, consult the main [Buttercup documentation](README.md) or check the [troubleshooting guide](CLAUDE.md#common-debugging-commands).
+
+
+[1]: https://aicyberchallenge.com/
+[2]: https://google.github.io/oss-fuzz/getting-started/new-project-guide/
@@ -138,3 +138,4 @@ kubectl logs -n crs <pod-name> -f
 - [AKS Deployment Guide](AKS_DEPLOYMENT.md) - Production deployment on Azure
 - [Contributing Guidelines](CONTRIBUTING.md) - Development workflow and standards
 - [Deployment Documentation](deployment/README.md) - Advanced deployment configuration
+- [Writing Custom Challenges](CUSTOM_CHALLENGES.md) - Custom project configuration and setup