This PR introduces a NodeLocal storage concept. Data is always processed on the local node, and only replicated to the azure file storage as needed.
A DaemonSet is introduced that syncs any corpus on the local node with the global corpus directory.
The following path structure is introduced:
/node_data/crs_scratch
/node_data/tasks_storage
/node_data/scratch
The first two directories is the equivalent of the previous usage of
/crs_scratch
/tasks_storage
The scratch is for basing temporary files and directories on the same filesystem as node-local version to be able to do atomic renames.
Further, ChallengeTask data is used normally on the /node_data/... paths, but is stored as /tasks_storage/{uuid}.tgz on the "remote" storage. This is to speed up copying.
Corpus and CrashSets are synced as is.
* Add zookeeper test file
* Update java function query.
* Add caller/callee queries
* Add example test for C with fixture
* Refactor challenge task creation in tests
---------
Co-authored-by: Boyan MILANOV <boyanmilanov@coder-boyanmilanov-aixcc-boyan.c.production-1-405717.internal>
* Use alternative type alias for BuildType type hints
* Remove unrelated test file
* check
---------
Co-authored-by: Boyan-MILANOV <boyan.milanov@trailofbits.com>
* Make build type an int enum
* Lint
* Fix protobuf
* Fix protobuf for linux
* Remove old enum and use the class from protobuf message
* Lint
* Fix tests
* Print build type name instead of int value in logs
* Try fix enum instanciation in scheduler
* Correctly get protobuf enum name for BuildType values using EnumTypeWrapper methods
* Fix import
---------
Co-authored-by: Boyan MILANOV <boyanmilanov@coder-boyanmilanov-aixcc-boyan.c.production-1-405717.internal>
Previously, if all harnesses had zero weight on would still be chosen.
This ensures only harnesses of weight >0.0 are considered and allows for
proper stop.
A minor change to the task cancellation was also made to ensure delete
all tasks message was acknowledged properly.
* Copy signoz deployment from example-crs-architecture
* Make ports local and remove unused clickhouse config
* Upload seed-gen LLM traces to SigNoz with OTel
* Add telemetry to patcher and seed-gen with common function
* Update README and lockfiles
* patcher: add tool to retrieve types
* program-model: fix codequery indexing
CodeQuery was run on the task without the patch applied, thus the
context was confusing the patcher, which was not able to generate a new
patch
Riccardo Schirone