This PR introduces a NodeLocal storage concept. Data is always processed on the local node, and only replicated to the azure file storage as needed.
A DaemonSet is introduced that syncs any corpus on the local node with the global corpus directory.
The following path structure is introduced:
/node_data/crs_scratch
/node_data/tasks_storage
/node_data/scratch
The first two directories is the equivalent of the previous usage of
/crs_scratch
/tasks_storage
The scratch is for basing temporary files and directories on the same filesystem as node-local version to be able to do atomic renames.
Further, ChallengeTask data is used normally on the /node_data/... paths, but is stored as /tasks_storage/{uuid}.tgz on the "remote" storage. This is to speed up copying.
Corpus and CrashSets are synced as is.
* Initial helm config
* Make sure all the fuzzer/builder/tracer.. have env vars
* Fixing env
* JanusGraph connect
* Runs up to patcher (without program db/seed-gen)
* Build bot with sidecar
* Docker sidecar to tracer, fuzzer and coverage bots
* Dind sidecar template
* Add bot-template
* Refactor, simplify
* Maps, not list fof volumes
* Enable all serviecs
* taskserver
* Use published images
* refactor
* dind for program-model and cleanup
* Remove unused envs
* Restore to main version
* gitignore values-override
* Single build-bot in test
* Support langfuse
* Cleanup
* Conditional langfuse secrets impl
* LiteLLM config
* JanusGraph
* Docker in docker sidecars
* Prevent double identical log messages
* Always use secrets if enabling langfuse
* Use NodePort locally and LoadBalancer in AKS
This is a first step towards an AKS-based system.
* Prepare for AKS (Storage/pull-secret)
- Renamed the pull secret to match terraform.
- Added AKS csi file storage
* updated charts
* Remove md file
* Remove program model mock variable
* Make use of templates for volumes
* Fix ghcr pull
* Cleanup - remove janusgraph and cassandra
Fixes an issue with the persistent-volumes.yaml rendering
* Remove janusgraph configmap
* Remove service account
* Remove unneeded config
* Drop install script as we don't need it
* Cleanup langfuse, add task-server auth
* litellm secrets
* Fix env/langfuse config propagation
* Adding a readme
* version
* Remove task server tgz
* update readme
* Remove redundant port
Riccardo Schirone