mirror of
https://github.com/trailofbits/buttercup
synced 2026-06-21 14:11:39 +00:00
Dan Guido
cc938e105f
* refactor: standardize packaging across all components - Standardize Python version to >=3.12,<3.13 for all components - Migrate from [project.optional-dependencies] to modern [dependency-groups] (PEP 735) - Standardize ruff line-length to 120 characters across all components - Add consistent project metadata: - AGPL-3.0 license field - Repository and Issues URLs - Improved, descriptive description fields - Fix email addresses to include .com domain This improves consistency, maintainability, and follows modern Python packaging best practices with uv/pip standards. * fix: address PR review comments - Move requires-python field to standard position (after license) in seed-gen - Update all components to use latest ruff version (>=0.12.8) - Ensure consistent dependency ordering across all components * refactor: standardize dependency pinning strategy Apply consistent dependency versioning across all components: - Use ~= (compatible release) for core dependencies: - Infrastructure: redis, pydantic, fastapi, uvicorn, sqlalchemy - AI/LLM: openai, langchain-community, langgraph-checkpoint - Parsing: tree-sitter, tree-sitter-language-pack - Web: requests, urllib3, pyyaml - Utils: python-dotenv, unidiff, argon2-cffi, pymongo, six - Keep >= for stable dev tools: - pytest, mypy, ruff, flake8 (want latest versions) - types-* packages (want latest type definitions) - rich, beautifulsoup4 (stable, backwards compatible) - Keep exact pins for known issues: - protobuf (narrow range for compatibility) - openlit==1.32.12 (documented issue with 1.33) - clusterfuzz==2.6.0 (complex, version-sensitive) This provides predictable builds with automatic patch updates while preventing unexpected breaking changes from major/minor version bumps. * feat: add project metadata for discoverability Add comprehensive metadata to all components: Keywords: - common: cybersecurity, crs, utilities, protobuf, redis, telemetry - fuzzer: fuzzing, oss-fuzz, libfuzzer, vulnerability-discovery, coverage - orchestrator: orchestration, task-management, scheduler, api, fastapi - patcher: patching, vulnerability-repair, llm, ai, code-generation - program-model: static-analysis, codequery, tree-sitter, semantic-analysis - seed-gen: test-generation, input-generation, fuzzing, seed-corpus, llm Classifiers: - Development Status :: 4 - Beta (all components) - License :: OSI Approved :: GNU Affero General Public License v3 - Programming Language :: Python :: 3.12 - Topic :: Security (all components) - Component-specific topics (Testing, AI, Distributed Computing, etc.) - Operating System :: POSIX :: Linux URLs: - Added Documentation URL pointing to README for all components This improves package discoverability, provides clear metadata for tools, and gives the project a more professional appearance. * Standardize tool configurations across all components - Add pytest.ini_options configuration to all components - Add coverage configuration with consistent exclude patterns - Standardize ruff configuration with target-version and lint rules - Fix missing readme field in fuzzer/pyproject.toml - Fix python-dotenv spacing inconsistency in seed-gen - Standardize all dev dependencies to use ~= operator for consistency * Fix trailing whitespace and line length issues - Remove trailing whitespace from tree-sitter query strings - Remove trailing whitespace from test output strings - Fix line length issues in logger.info() calls by splitting format strings - Fix line length in datetime formatting by extracting variables - Split long Pydantic Field descriptions and docstrings - Leave test data strings unchanged to avoid breaking tests * Fix dependency resolution issues - Update argon2-cffi from ~=21.0.0 to ~=21.3.0 (21.0.x doesn't exist on PyPI) - Update langgraph-checkpoint from ~=2.0.25 to ~=2.1.0 to match langgraph requirements - Standardize spacing around ~= operators in all dependency specifications - All components now successfully resolve dependencies with uv * Apply ruff auto-fixes across project - Fix import sorting (I001) in fuzzer, orchestrator, and patcher - Update to PEP 585 type annotations (List->list, Dict->dict, etc.) - Update to PEP 604 union syntax (Optional[X] -> X | None) - Remove unnecessary UTF-8 encoding declarations - Remove redundant file open modes - Modernize type annotations throughout the codebase Remaining issues are primarily line length (E501) which require manual review * Fix line length issues in program-model component - Break up long Java code strings in test assertions using implicit concatenation - Split long constructor and method definitions across multiple lines - Add noqa: E501 comment for 10,977 character struct definition test data - All program-model line length issues resolved * fix: revert protobuf enum type annotations to Optional Protobuf enums (EnumTypeWrapper) don't support the | operator for type unions. The ruff UP035 rule converted Optional[ProtobufEnum] to ProtobufEnum | None, but this causes TypeError at runtime. Reverted these specific changes while keeping the modern type union syntax for regular Python types. * chore: add ruff protection for protobuf enum type annotations - Configure ruff to ignore UP045 rule in test_submissions.py - Add inline noqa comments to document why Optional is needed - Protobuf enums (EnumTypeWrapper) don't support the | operator - This prevents future automated fixes from breaking the code * fix: modernize Python syntax and fix formatting issues - Convert printf-style formatting to f-strings (UP031) - Remove trailing whitespace from blank lines (W293) - Use PEP 695 generic class syntax for Python 3.12+ (UP046) - Use PEP 695 type alias syntax with 'type' keyword (UP040) These changes modernize the codebase to use Python 3.12+ features and fix formatting inconsistencies detected by ruff. * fix: resolve undefined MsgType reference after PEP 695 conversion When converting to PEP 695 generic class syntax, the MsgType TypeVar was removed but was still referenced in overloaded method signatures. Changed the generic fallback overload to use Message directly. * fix: resolve line length violations across entire codebase Applied Black formatter and manual fixes to resolve E501 line length violations: - Fixed 178 line length issues across common, fuzzer, orchestrator, patcher, and program-model components - Used Black formatter for automatic reformatting where possible - Manually split long strings, function calls, and complex expressions - All files now comply with 120-character line limit 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: revert PEP 695 type alias syntax in node_local.py The PEP 695 syntax (type X = Y) creates TypeAliasType objects that cannot be used as constructors at runtime. Since node_local.py uses NodeLocalPath and RemotePath as constructors (e.g., NodeLocalPath(path)), we must use the old TypeAlias syntax to maintain runtime functionality. Added noqa comments to prevent ruff from attempting to modernize these aliases in the future. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * chore: configure ruff to ignore UP040 for node_local.py Added per-file configuration to prevent ruff from attempting to convert TypeAlias annotations to PEP 695 syntax in node_local.py. This protects the runtime functionality that relies on these type aliases being usable as constructors. Also removed redundant inline noqa comments since the ignore is now configured at the project level. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: correct MsgType reference in static method decorator The _ensure_group_name static method decorator was incorrectly referencing MsgType in the wrapper function signature. Since MsgType is a class-level type parameter and not accessible in static method scope, changed it to Message which is the appropriate bound type. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * docs: add explanatory comment for Message type in decorator Added a comment explaining why we must use Message instead of MsgType in the _ensure_group_name decorator's wrapper function. This prevents future confusion and protects against accidental "fixes" that would break the code. The MsgType parameter is a class-level type variable that's not in scope within the static method decorator context. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
Buttercup Common utilities
Protobufs
The protos directory contains the protobuf definitions for various messages used by the Buttercup system.
Buttercup Util utility
The buttercup-util script is a utility for interacting with various components of the Buttercup CRS.
$ buttercup-util --help
usage: buttercup-util [-h] [--redis_url str] [--log_level str] {send_queue,read_queue,list_queues,delete_queue,add_harness,add_build,read_harnesses,read_builds} ...
options:
-h, --help show this help message and exit
--redis_url str Redis URL (default: redis://localhost:6379)
--log_level str Log level (default: info)
subcommands:
{send_queue,read_queue,list_queues,delete_queue,add_harness,add_build,read_harnesses,read_builds}
send_queue
read_queue
list_queues
delete_queue
add_harness
add_build
read_harnesses
read_builds
Send messages to a specific queue
$ buttercup-util send_queue orchestrator_download_tasks_queue ./examples/task_download.txt
2025-03-12 10:49:46,342 - buttercup.common.util_cli - INFO - Reading TaskDownload message from file 'examples/task_download.txt'
2025-03-12 10:49:46,342 - buttercup.common.util_cli - INFO - Pushing message to queue 'orchestrator_download_tasks_queue': task {
message_time: 1739917788000
task_id: "my-task-id"
task_type: TASK_TYPE_DELTA
sources {
sha256: "c516e2b73f58fe163be48f5bc0ca36995ee100c752e30883f9acaa0a95ca2bb6"
url: "https://challengesact.blob.core.windows.net/challenges/c516e2b73f58fe163be48f5bc0ca36995ee100c752e30883f9acaa0a95ca2bb6.tar.gz?se=2025-08-18T22%3A29%3A44Z&sp=r&sv=2022-11-02&sr=b&sig=7lj49Z6vXsFuKp4DqVrVVMwHU4xEAQJ%2BSCZ7BAQnbvY%3D"
}
sources {
sha256: "910913fd13eb2e7cb7ca9a39fce4cc753d54579c938a5c60d478788101fdde3e"
source_type: SOURCE_TYPE_FUZZ_TOOLING
url: "https://challengesact.blob.core.windows.net/challenges/910913fd13eb2e7cb7ca9a39fce4cc753d54579c938a5c60d478788101fdde3e.tar.gz?se=2025-08-18T22%3A29%3A47Z&sp=r&sv=2022-11-02&sr=b&sig=M6JoI0pGccbSARTqVLm23yQZUbUwsQsFyBpRMoADnYc%3D"
}
sources {
sha256: "04ffd1402d868846d6812112c4bc2ec50722aa1adfaf02aab7233ad20bd1b495"
source_type: SOURCE_TYPE_DIFF
url: "https://challengesact.blob.core.windows.net/challenges/04ffd1402d868846d6812112c4bc2ec50722aa1adfaf02aab7233ad20bd1b495.tar.gz?se=2025-08-18T22%3A29%3A42Z&sp=r&sv=2022-11-02&sr=b&sig=M63mfyTls1CJhxelj%2BdtGmmO9fIVimybM6yqOMCkRac%3D"
}
deadline: 1739932188000
project_name: "libpng"
focus: "example-libpng"
}
Read an entire queue
$ buttercup-util read_queue orchestrator_download_tasks_queue
[...]
Or, if you want to simulate a consumer in a consumer group popping an element out of the queue:
$ buttercup-util read_queue orchestrator_download_tasks_queue --group_name orchestrator_group
Add an harness to the fuzzer map
$ buttercup-util add_harness ./examples/weighted_harness.txt
2025-03-12 10:53:10,256 - buttercup.common.util_cli - INFO - Added harness weight for libpng | libpng_read_fuzzer | my-task-id
$ buttercup-util read_harnesses
weight: 1.0
package_name: "libpng"
harness_name: "libpng_read_fuzzer"
task_id: "my-task-id"
2025-03-12 10:55:35,483 - buttercup.common.util_cli - INFO - Done
Add a build to the build map
$ buttercup-util add_build ./examples/build.txt
2025-03-12 10:55:07,273 - buttercup.common.util_cli - INFO - Added build for my-task-id | fuzzer | address
$ buttercup-util read_builds my-task-id fuzzer
engine: "libfuzzer"
sanitizer: "address"
task_dir: "/crs_scratch/my-task-id/"
task_id: "my-task-id"
build_type: "FUZZER"
apply_diff: true
2025-03-12 10:55:20,298 - buttercup.common.util_cli - INFO - Done