* Add optional tmpfs storage for fuzzing corpus
Add support for storing the node-local fuzzing corpus in tmpfs (RAM-based
filesystem) for improved I/O performance during fuzzing operations.
Key changes:
- Add CORPUS_TMPFS_PATH environment variable to configure tmpfs location
- Add cross-filesystem safe operations (copy+delete fallback for EXDEV)
- Update Corpus class to use tmpfs path when enabled while maintaining
correct remote path calculation for rsync synchronization
- Add Helm chart configuration using /dev/shm (requires no host setup)
- Update fuzzer-bot, coverage-bot, merger-bot, seed-gen deployments
The feature is disabled by default. Enable by setting
global.volumes.corpusTmpfs.enabled=true in values.yaml.
Uses hostPath to /dev/shm/buttercup-corpus which is available on
Linux systems without requiring any Kubernetes host configuration.
The subprocess mock was missing __enter__ and __exit__ configuration,
causing tests to hang when ChallengeTask._run_cmd used
`with subprocess.Popen(...) as process:` context manager syntax.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* style: apply ruff auto-fixes and formatting across entire codebase
Applied safe auto-fixes from ruff v0.12.9 with --select ALL to improve code quality:
- Reorder imports (stdlib → third-party → local)
- Use modern type hints (collections.abc.Generator instead of typing.Generator)
- Add trailing commas for better diffs
- Format multi-line function parameters for readability
- Add strict=False to zip() calls for explicit behavior
- Simplify redundant elif to if after return statements
- Consistent code formatting with ruff format
These are all mechanical, non-controversial changes that improve code consistency
without altering functionality. Changes affect 180 files across all modules:
common, fuzzer, orchestrator, patcher, program-model, and seed-gen.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* re-applt ruff after merge
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Michael D Brown <michael.brown@trailofbits.com>
* fix: standardize packaging configuration across components
- Fix fuzzer package name from 'fuzzing-infra' to 'fuzzer' for consistency
- Add missing README field to common/pyproject.toml
- Standardize seed-gen script naming to follow buttercup-* convention
- Remove redundant flake8 dependency from orchestrator and patcher (ruff handles linting)
- Update lock files to reflect dependency changes
- Auto-format code in common and seed-gen components
* refactor: standardize dev dependencies across all components
- Add pytest-cov to all components for consistent test coverage reporting
- Standardize types-redis version to 4.6.0 across all components
- Standardize types-requests version to 2.32.0 where used
- Remove unused pytest-asyncio from orchestrator and patcher (no async tests found)
- Remove unused types-python-dateutil from patcher (only used in orchestrator)
- Keep component-specific dependencies where justified:
- orchestrator: pytest-xdist for parallel testing, API testing tools
- patcher: pytest-xdist for parallel testing
- common: types-PyYAML, dirty-equals for testing
- seed-gen: tqdm for progress bars
All components now have consistent base dev dependencies with optional extras
as needed. This improves maintainability and reduces confusion.
* fix: update deployment configs for renamed seed-gen command
- Update docker-compose to use buttercup-seed-gen command
- Update Kubernetes/Helm deployment to use buttercup-seed-gen command
- These are critical fixes to ensure deployments work with the renamed command
* refactor: standardize packaging across all components
- Standardize Python version to >=3.12,<3.13 for all components
- Migrate from [project.optional-dependencies] to modern [dependency-groups] (PEP 735)
- Standardize ruff line-length to 120 characters across all components
- Add consistent project metadata:
- AGPL-3.0 license field
- Repository and Issues URLs
- Improved, descriptive description fields
- Fix email addresses to include .com domain
This improves consistency, maintainability, and follows modern Python
packaging best practices with uv/pip standards.
* fix: address PR review comments
- Move requires-python field to standard position (after license) in seed-gen
- Update all components to use latest ruff version (>=0.12.8)
- Ensure consistent dependency ordering across all components
* refactor: standardize dependency pinning strategy
Apply consistent dependency versioning across all components:
- Use ~= (compatible release) for core dependencies:
- Infrastructure: redis, pydantic, fastapi, uvicorn, sqlalchemy
- AI/LLM: openai, langchain-community, langgraph-checkpoint
- Parsing: tree-sitter, tree-sitter-language-pack
- Web: requests, urllib3, pyyaml
- Utils: python-dotenv, unidiff, argon2-cffi, pymongo, six
- Keep >= for stable dev tools:
- pytest, mypy, ruff, flake8 (want latest versions)
- types-* packages (want latest type definitions)
- rich, beautifulsoup4 (stable, backwards compatible)
- Keep exact pins for known issues:
- protobuf (narrow range for compatibility)
- openlit==1.32.12 (documented issue with 1.33)
- clusterfuzz==2.6.0 (complex, version-sensitive)
This provides predictable builds with automatic patch updates while
preventing unexpected breaking changes from major/minor version bumps.
* feat: add project metadata for discoverability
Add comprehensive metadata to all components:
Keywords:
- common: cybersecurity, crs, utilities, protobuf, redis, telemetry
- fuzzer: fuzzing, oss-fuzz, libfuzzer, vulnerability-discovery, coverage
- orchestrator: orchestration, task-management, scheduler, api, fastapi
- patcher: patching, vulnerability-repair, llm, ai, code-generation
- program-model: static-analysis, codequery, tree-sitter, semantic-analysis
- seed-gen: test-generation, input-generation, fuzzing, seed-corpus, llm
Classifiers:
- Development Status :: 4 - Beta (all components)
- License :: OSI Approved :: GNU Affero General Public License v3
- Programming Language :: Python :: 3.12
- Topic :: Security (all components)
- Component-specific topics (Testing, AI, Distributed Computing, etc.)
- Operating System :: POSIX :: Linux
URLs:
- Added Documentation URL pointing to README for all components
This improves package discoverability, provides clear metadata for tools,
and gives the project a more professional appearance.
* Standardize tool configurations across all components
- Add pytest.ini_options configuration to all components
- Add coverage configuration with consistent exclude patterns
- Standardize ruff configuration with target-version and lint rules
- Fix missing readme field in fuzzer/pyproject.toml
- Fix python-dotenv spacing inconsistency in seed-gen
- Standardize all dev dependencies to use ~= operator for consistency
* Fix trailing whitespace and line length issues
- Remove trailing whitespace from tree-sitter query strings
- Remove trailing whitespace from test output strings
- Fix line length issues in logger.info() calls by splitting format strings
- Fix line length in datetime formatting by extracting variables
- Split long Pydantic Field descriptions and docstrings
- Leave test data strings unchanged to avoid breaking tests
* Fix dependency resolution issues
- Update argon2-cffi from ~=21.0.0 to ~=21.3.0 (21.0.x doesn't exist on PyPI)
- Update langgraph-checkpoint from ~=2.0.25 to ~=2.1.0 to match langgraph requirements
- Standardize spacing around ~= operators in all dependency specifications
- All components now successfully resolve dependencies with uv
* Apply ruff auto-fixes across project
- Fix import sorting (I001) in fuzzer, orchestrator, and patcher
- Update to PEP 585 type annotations (List->list, Dict->dict, etc.)
- Update to PEP 604 union syntax (Optional[X] -> X | None)
- Remove unnecessary UTF-8 encoding declarations
- Remove redundant file open modes
- Modernize type annotations throughout the codebase
Remaining issues are primarily line length (E501) which require manual review
* Fix line length issues in program-model component
- Break up long Java code strings in test assertions using implicit concatenation
- Split long constructor and method definitions across multiple lines
- Add noqa: E501 comment for 10,977 character struct definition test data
- All program-model line length issues resolved
* fix: revert protobuf enum type annotations to Optional
Protobuf enums (EnumTypeWrapper) don't support the | operator for type unions.
The ruff UP035 rule converted Optional[ProtobufEnum] to ProtobufEnum | None,
but this causes TypeError at runtime. Reverted these specific changes while
keeping the modern type union syntax for regular Python types.
* chore: add ruff protection for protobuf enum type annotations
- Configure ruff to ignore UP045 rule in test_submissions.py
- Add inline noqa comments to document why Optional is needed
- Protobuf enums (EnumTypeWrapper) don't support the | operator
- This prevents future automated fixes from breaking the code
* fix: modernize Python syntax and fix formatting issues
- Convert printf-style formatting to f-strings (UP031)
- Remove trailing whitespace from blank lines (W293)
- Use PEP 695 generic class syntax for Python 3.12+ (UP046)
- Use PEP 695 type alias syntax with 'type' keyword (UP040)
These changes modernize the codebase to use Python 3.12+ features
and fix formatting inconsistencies detected by ruff.
* fix: resolve undefined MsgType reference after PEP 695 conversion
When converting to PEP 695 generic class syntax, the MsgType TypeVar
was removed but was still referenced in overloaded method signatures.
Changed the generic fallback overload to use Message directly.
* fix: resolve line length violations across entire codebase
Applied Black formatter and manual fixes to resolve E501 line length violations:
- Fixed 178 line length issues across common, fuzzer, orchestrator, patcher, and program-model components
- Used Black formatter for automatic reformatting where possible
- Manually split long strings, function calls, and complex expressions
- All files now comply with 120-character line limit
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* fix: revert PEP 695 type alias syntax in node_local.py
The PEP 695 syntax (type X = Y) creates TypeAliasType objects that cannot
be used as constructors at runtime. Since node_local.py uses NodeLocalPath
and RemotePath as constructors (e.g., NodeLocalPath(path)), we must use
the old TypeAlias syntax to maintain runtime functionality.
Added noqa comments to prevent ruff from attempting to modernize these
aliases in the future.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* chore: configure ruff to ignore UP040 for node_local.py
Added per-file configuration to prevent ruff from attempting to convert
TypeAlias annotations to PEP 695 syntax in node_local.py. This protects
the runtime functionality that relies on these type aliases being usable
as constructors.
Also removed redundant inline noqa comments since the ignore is now
configured at the project level.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* fix: correct MsgType reference in static method decorator
The _ensure_group_name static method decorator was incorrectly
referencing MsgType in the wrapper function signature. Since MsgType
is a class-level type parameter and not accessible in static method
scope, changed it to Message which is the appropriate bound type.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* docs: add explanatory comment for Message type in decorator
Added a comment explaining why we must use Message instead of MsgType
in the _ensure_group_name decorator's wrapper function. This prevents
future confusion and protects against accidental "fixes" that would
break the code.
The MsgType parameter is a class-level type variable that's not in
scope within the static method decorator context.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* Fix integration-test run
* use the proper oss-fuzz-aixcc commit to run integration-test
* determine OSS_FUZZ_CONTAINER_ORG at runtime
* use `git diff` in the patcher to create the patch to fix a problem
with patches affecting non-newline terminated files
* fix parsing
* fix program-model lint
* Modify CRS to work on MacOS/ARM
* Use LibUCL for testing
* use example-libpng
* fix linting
* common: fix tests
* Fix integration-test run
* use the proper oss-fuzz-aixcc commit to run integration-test
* determine OSS_FUZZ_CONTAINER_ORG at runtime
* use `git diff` in the patcher to create the patch to fix a problem
with patches affecting non-newline terminated files
* fix parsing
* fix program-model lint
This will differentiate between vulnerabilities in different lines of
the same function, so that they all go to the tracer-bot for validation.
In this way, if there is a DELTA mode challenge and one vuln is
present both before and after the delta, but another one is introduced
by the delta, we are able to validate them separately and actually find
the vulnerability that is introduced by the delta (and not discard it in
the fuzzer-bot due to being already present in the CrashSet).
It means the tracer-bot will receive slightly more crashes to validate,
but still a reasonable number.
Co-authored-by: Michael D Brown <michael.brown@trailofbits.com>
* Don't process PoVs that exceed the max size
* WIP enforce seed and PoV size limits in seed-gen
* Revert "Don't process PoVs that exceed the max size"
This reverts commit 8cee343160ba299dccd12284f65934efda63b2ef.
* Enforce max PoV size in fuzzer
* Set max seed size to 64 KiB
* Fixup
* Pass copy_corpus_max_size to Corpus and InputDir
* Fixup
* Use an internal_patch_id instead of indices
This is the first step in being able to merge sets of PoVs and test
patches against all PoVs within a task.
* Discard redundant builds
* Initial PoV-merging strategy
Still not optimal in terms of SARIF-matching/bundling etc
* Appears to be working version of merging including bundle and sarif handling
* Make tests pass
* Update integration test steps
* Fixes and cleanup from review
* Removed additional request for patched builds
* Refactored some loops into find-style functions to simplify
* Inline small function used once
* Refactors for increased robustness and readability including additional testing
* SARIF matching - additional tests and refactor
* Add enumerate_task_submissions
* Refactor and simplify tests using a builder
Cleanup unused code
* Fix read_submissions to use CrashWithId
* Improvements based on review
* Cache final states of PoV reproduce (#909)
As these never change we can limit the load on redis by caching the
results.
* Merge SubmissionEntries based on patches (#910)
* Cache final states of PoV reproduce
As these never change we can limit the load on redis by caching the
results.
* Merge SubmissionEntries based on patches
If a PoV in another entry is mitigated by the current entry's patch,
merge the entries as athey should be considered the same
ChallengeVulnerability.
* Add positional argument
* Hold of submitting a patch while evaluating
Check each already submitted patch before submitting a new one for the
same task. If any of the already submitted patches mitigates any PoV in
the current SubmissionEntry - do not submit this. It will be merged
later on.
* Additional logging, truncate ids
* Only request patch if no submitted patch mitigates
Before we request a new patch, we check each of the already submitted
patches to see if any of them already mitigates the PoVs in the current
SubmissionEntry. If they do, this will be merged at a later stage.
* PR feedback
* Use an internal_patch_id instead of indices
This is the first step in being able to merge sets of PoVs and test
patches against all PoVs within a task.
* Discard redundant builds
* Compare relevant fields
* common: provide unified_language property in ProjectYaml
* common: use also cpp
* Applying fixup to missed location
* codequery: fix how unified_language is used
* program-model: fix filter_project_context for new Language enum
* program-model: fix tests
* fix program model tests
---------
Co-authored-by: Michael D. Brown <michael.brown@trailofbits.com>
* WIP
* WIP
* WIP still needs testing and unit tests
* Fix typo
* Update / add tests
* use correct id
* Add backend test
* fix new test
* missing base calss added
* Fix side effects
If files are renamed or deleted while computing the accumulated size it
previously failed. Now it will just move on to the next. The local
corpus size isn't an exact value and changes constantly so if we miss
one file it is not a problem.
This PR introduces a NodeLocal storage concept. Data is always processed on the local node, and only replicated to the azure file storage as needed.
A DaemonSet is introduced that syncs any corpus on the local node with the global corpus directory.
The following path structure is introduced:
/node_data/crs_scratch
/node_data/tasks_storage
/node_data/scratch
The first two directories is the equivalent of the previous usage of
/crs_scratch
/tasks_storage
The scratch is for basing temporary files and directories on the same filesystem as node-local version to be able to do atomic renames.
Further, ChallengeTask data is used normally on the /node_data/... paths, but is stored as /tasks_storage/{uuid}.tgz on the "remote" storage. This is to speed up copying.
Corpus and CrashSets are synced as is.
* common: add workdir_from_dockerfile API in ChallengeTask
* program-model: mount local source directory when indexing
* program-model: configure graphdb_url from settings
* Get diff from ChallengeTask instead of mock
* Remove mock diff and don't pass WeightedHarness to Task
* Address review feedback
* Update seed-gen env var in CI
* multiple builds per build type based on sanitizer
* update clients for now
* fixes
* formatting
* additional method
* add back name for now
* build builds of various types
* implement tracer bot
* use task loop with list
* fix users
* fix
* add max tries to tracer
* add test
* formatting
* fix field type
* fix default
* coverage_builds
* annotation
* restore old annotation
* Create the TaskMeta structure in the task directory
it contains information about project name and focus.
* lint
* Address review
* Restore name property
* common: move diff apply into ChallengeTask
* common: do not raise exceptions if no patch
* common: use raw string for mocked patch
* common: enforce ordering in get_diffs
Henrik Brodin