This PR introduces a NodeLocal storage concept. Data is always processed on the local node, and only replicated to the azure file storage as needed.
A DaemonSet is introduced that syncs any corpus on the local node with the global corpus directory.
The following path structure is introduced:
/node_data/crs_scratch
/node_data/tasks_storage
/node_data/scratch
The first two directories is the equivalent of the previous usage of
/crs_scratch
/tasks_storage
The scratch is for basing temporary files and directories on the same filesystem as node-local version to be able to do atomic renames.
Further, ChallengeTask data is used normally on the /node_data/... paths, but is stored as /tasks_storage/{uuid}.tgz on the "remote" storage. This is to speed up copying.
Corpus and CrashSets are synced as is.
* Add zookeeper test file
* Update java function query.
* Add caller/callee queries
* Add example test for C with fixture
* Refactor challenge task creation in tests
---------
Co-authored-by: Boyan MILANOV <boyanmilanov@coder-boyanmilanov-aixcc-boyan.c.production-1-405717.internal>
* Copy signoz deployment from example-crs-architecture
* Make ports local and remove unused clickhouse config
* Upload seed-gen LLM traces to SigNoz with OTel
* Add telemetry to patcher and seed-gen with common function
* Update README and lockfiles
* patcher: add tool to retrieve types
* program-model: fix codequery indexing
CodeQuery was run on the task without the patch applied, thus the
context was confusing the patcher, which was not able to generate a new
patch
* common: add workdir_from_dockerfile API in ChallengeTask
* program-model: mount local source directory when indexing
* program-model: configure graphdb_url from settings
* program-model/tree-sitter: refactor classes
* program-model/tree-sitter: change how paths are considered
This allows to parse also code in oss-fuzz (e.g. injected fuzz targets
not in upstream project)
* common: change name format for challenge commit
* program-model/codequery: make it persistent and nice
* program-model: disable janus & co for now
* patcher: make use of codequery instead of just tree-sitter
* program-model: add integration tests for codequery
* program-model/codequery: add support for different languages
* patcher: install codequery in patcher's container
* ci: fix codequery install
* program-model: ensure cqdb storage in /crs_scratch
* program-model: ensure the cqdb storage is available only at the end
* Add skeleton for api call
* Modify encoding and decoding nodes/edges
* Add change encoding from base64 to hex
* reformat
* Target libpng function in example
* remove old project name
* cli fixes
* Add get_function via byte offsets
* refactor
---------
Co-authored-by: Ian Smith <ian.smith@trailofbits.com>
* Create the TaskMeta structure in the task directory
it contains information about project name and focus.
* lint
* Address review
* Restore name property
* Enforce python >=3.10 and use ignore_cleanup_errors in TemporaryDirectory
* have an upper bound on requires-python
* update uv.locks
---------
Co-authored-by: Riccardo Schirone <riccardo.schirone@trailofbits.com>
* Add documentation
* Add janusgraph custom configuration. Simply graph creation.
* Add properties to edges. Index on uri property.
* reformat
* Create libpng graph in a more reasonable amount of time.
* Update README.md
* Wrap indexing into function
* updating...
* update proto
* fix command
* Update settings for serve and processing indexing
* Add dockerfile to docker container
* Add trigger script
* Fix error with loading graphml file
* reformat
* Remove resource constraints for program model
* Change permissions of temp dir
* Update readme
* Update dockerignore
---------
Co-authored-by: Evan Downing <2077950+evandowning@users.noreply.github.com>
* add program model uv
* add tool to common
* remove oss fuzz tool
* add docker
* implement build base image
* fix ret check
* add builder for subimage
* remove old tool
* hack to avoid dupped kzip files
* fixes... need to work out LD hack
* fix find
* justfile for building kythe
* update dockerfile
* add upload tool
* update protobuf
* move into compose
* fix merge invocation
* fix up invoc
* bump kythe
* reformat
* format
* format
* remove unused
* add lint and change version
* fixes
* exclude
* fix formatting:
* dev deps
* add dep
* exclude gen
* fix
Boyan MILANOV