This PR introduces a NodeLocal storage concept. Data is always processed on the local node, and only replicated to the azure file storage as needed.
A DaemonSet is introduced that syncs any corpus on the local node with the global corpus directory.
The following path structure is introduced:
/node_data/crs_scratch
/node_data/tasks_storage
/node_data/scratch
The first two directories is the equivalent of the previous usage of
/crs_scratch
/tasks_storage
The scratch is for basing temporary files and directories on the same filesystem as node-local version to be able to do atomic renames.
Further, ChallengeTask data is used normally on the /node_data/... paths, but is stored as /tasks_storage/{uuid}.tgz on the "remote" storage. This is to speed up copying.
Corpus and CrashSets are synced as is.
* Add zookeeper test file
* Update java function query.
* Add caller/callee queries
* Add example test for C with fixture
* Refactor challenge task creation in tests
---------
Co-authored-by: Boyan MILANOV <boyanmilanov@coder-boyanmilanov-aixcc-boyan.c.production-1-405717.internal>
* program-model/tree-sitter: refactor classes
* program-model/tree-sitter: change how paths are considered
This allows to parse also code in oss-fuzz (e.g. injected fuzz targets
not in upstream project)
* common: change name format for challenge commit
* program-model/codequery: make it persistent and nice
* program-model: disable janus & co for now
* patcher: make use of codequery instead of just tree-sitter
* program-model: add integration tests for codequery
* program-model/codequery: add support for different languages
* patcher: install codequery in patcher's container
* ci: fix codequery install
* program-model: ensure cqdb storage in /crs_scratch
* program-model: ensure the cqdb storage is available only at the end
Evan Downing