This PR introduces a NodeLocal storage concept. Data is always processed on the local node, and only replicated to the azure file storage as needed.
A DaemonSet is introduced that syncs any corpus on the local node with the global corpus directory.
The following path structure is introduced:
/node_data/crs_scratch
/node_data/tasks_storage
/node_data/scratch
The first two directories is the equivalent of the previous usage of
/crs_scratch
/tasks_storage
The scratch is for basing temporary files and directories on the same filesystem as node-local version to be able to do atomic renames.
Further, ChallengeTask data is used normally on the /node_data/... paths, but is stored as /tasks_storage/{uuid}.tgz on the "remote" storage. This is to speed up copying.
Corpus and CrashSets are synced as is.
* Use alternative type alias for BuildType type hints
* Remove unrelated test file
* check
---------
Co-authored-by: Boyan-MILANOV <boyan.milanov@trailofbits.com>
* Make build type an int enum
* Lint
* Fix protobuf
* Fix protobuf for linux
* Remove old enum and use the class from protobuf message
* Lint
* Fix tests
* Print build type name instead of int value in logs
* Try fix enum instanciation in scheduler
* Correctly get protobuf enum name for BuildType values using EnumTypeWrapper methods
* Fix import
---------
Co-authored-by: Boyan MILANOV <boyanmilanov@coder-boyanmilanov-aixcc-boyan.c.production-1-405717.internal>
Previously, if all harnesses had zero weight on would still be chosen.
This ensures only harnesses of weight >0.0 are considered and allows for
proper stop.
A minor change to the task cancellation was also made to ensure delete
all tasks message was acknowledged properly.
* Copy signoz deployment from example-crs-architecture
* Make ports local and remove unused clickhouse config
* Upload seed-gen LLM traces to SigNoz with OTel
* Add telemetry to patcher and seed-gen with common function
* Update README and lockfiles
* Calculate function coverage fraction
Note: function isn't tested yet
* Store function coverage in Redis
* Sample function off coverage map in seed-gen
* Conditionally update function coverage
* Try multiple definition lookup heuristics
* Log function probability and coverage fraction
* Fix calculate_function_probabilities
* Fix docstring
* Implements task cancelation
- Cancel one or all tasks.
- delete_task.sh script for invoking
- refactored to keep deleted tasks in a separate set, update tasks as
needed from registry instead of updating the stored tasks.
* Refactor cancellation, update use in vulnerabilities
* Cleanup cancellation
* Remove package_name from protomsg
* Fix test case
* fix tests
* PR feedback
* common: add workdir_from_dockerfile API in ChallengeTask
* program-model: mount local source directory when indexing
* program-model: configure graphdb_url from settings
* program-model/tree-sitter: refactor classes
* program-model/tree-sitter: change how paths are considered
This allows to parse also code in oss-fuzz (e.g. injected fuzz targets
not in upstream project)
* common: change name format for challenge commit
* program-model/codequery: make it persistent and nice
* program-model: disable janus & co for now
* patcher: make use of codequery instead of just tree-sitter
* program-model: add integration tests for codequery
* program-model/codequery: add support for different languages
* patcher: install codequery in patcher's container
* ci: fix codequery install
* program-model: ensure cqdb storage in /crs_scratch
* program-model: ensure the cqdb storage is available only at the end
* common: ignore errors when trying to restore a challenge
* patcher: allow SWE agent to request new context
* patcher: some other refactoring re code snippets
* Get function body from program-model for seed-explore
* Add harness finding fallback
Also add language field to project.yaml class
* Update test to have new TaskMeta param
* Install ripgrep in CI
* Search oss-fuzz project dir and add test cases
* Get diff from ChallengeTask instead of mock
* Remove mock diff and don't pass WeightedHarness to Task
* Address review feedback
* Update seed-gen env var in CI
* patcher: use pydantic basemodel
* patcher: restructure to use Command
* patcher: remove mock mode because not needed anymore
* remove also env var in docker compose for mock mode
* patcher: remove filtering for now
* patcher: remove | None to fields with default
* Add vuln discovery eval script
* Handle multiple markdown blocks in response
* Add langfuse metadata
* Add timestamp to output dirs
* Make abstract class
* Move vuln discovery eval script
* Make VulnDiscovery class
* Add eval script to test current vuln discovery implementation
* Switch to Claude 3.5 Sonnet for seed-gen
* Update vuln discovery prompts
* Appease ruff
* Update seed-gen for changes from main
* Fix ReproduceMultiple
* multiple builds per build type based on sanitizer
* update clients for now
* fixes
* formatting
* additional method
* add back name for now
* build builds of various types
* implement tracer bot
* use task loop with list
* fix users
* fix
* add max tries to tracer
* add test
* formatting
* fix field type
* fix default
* coverage_builds
* annotation
* restore old annotation
* Create the TaskMeta structure in the task directory
it contains information about project name and focus.
* lint
* Address review
* Restore name property
* patcher: improve the patcher by being more flexible
* patcher: use ChallengeTask.restore to retry patches
* patcher: simplify a bit the create_upatch
* Implement actual build request creation
Removes the mock and adds functionality to parse the project yaml.
* Move the ProjectYaml to common
* lint
* Pass in the correct name
* pr feedback
* Tmpfix uvlock
* Enforce python >=3.10 and use ignore_cleanup_errors in TemporaryDirectory
* have an upper bound on requires-python
* update uv.locks
---------
Co-authored-by: Riccardo Schirone <riccardo.schirone@trailofbits.com>
* Cleanup of group names for consumers
We only really need distinct consumer group names if there are multiple
consumers. Most consumers are still the orchestrator components so I've
simplified things a bit and removed group names that weren't needed.
* fmt
* Add documentation
* Add janusgraph custom configuration. Simply graph creation.
* Add properties to edges. Index on uri property.
* reformat
* Create libpng graph in a more reasonable amount of time.
* Update README.md
* Wrap indexing into function
* updating...
* update proto
* fix command
* Update settings for serve and processing indexing
* Add dockerfile to docker container
* Add trigger script
* Fix error with loading graphml file
* reformat
* Remove resource constraints for program model
* Change permissions of temp dir
* Update readme
* Update dockerignore
---------
Co-authored-by: Evan Downing <2077950+evandowning@users.noreply.github.com>
* apply diff in builder bot
* fix dockerfile seed-gen
* format
* apply the diffs
* fix builds
* store if diff was applied
* Revert "format"
This reverts commit f07a806ebd98f16409f0c3676aefc50441562239.
* common: move diff apply into ChallengeTask
* common: do not raise exceptions if no patch
* common: use raw string for mocked patch
* common: enforce ordering in get_diffs
Riccardo Schirone