Commit Graph

137 Commits

Author SHA1 Message Date
Riccardo Schirone eb66060ac3 ci: run mypy on components (#132) 2025-04-09 09:56:16 +02:00
Riccardo Schirone 545e8526ef program-model: make sure challenge is rw (#297) 2025-03-28 11:06:33 +01:00
Ronald Eytchison 5e089cf6c4 Upload LLM OTel data to Signoz (#275)
* Copy signoz deployment from example-crs-architecture

* Make ports local and remove unused clickhouse config

* Upload seed-gen LLM traces to SigNoz with OTel

* Add telemetry to patcher and seed-gen with common function

* Update README and lockfiles
2025-03-28 11:04:45 +01:00
Evan Downing 6129f37c1b Add tests for java challenges (#285) 2025-03-27 14:53:10 -04:00
Riccardo Schirone 758e831063 patcher: use codequery.get_types and fix context (#287)
* patcher: add tool to retrieve types

* program-model: fix codequery indexing

CodeQuery was run on the task without the patch applied, thus the
context was confusing the patcher, which was not able to generate a new
patch
2025-03-27 17:11:14 +01:00
Evan Downing 5d60b606b7 Add retrieving types in codequery (#260)
* Add retrieving types in codequery

* Add fuzzy search option (#271)
2025-03-27 08:23:01 -04:00
Evan Downing 557dbb75b1 Update readme for java testing (#269) 2025-03-26 14:33:49 -04:00
Evan Downing ce9976867a Add environment variable to disable graphdb. (#253) 2025-03-26 07:23:26 -04:00
Evan Downing cc9e863649 Decode property to string (#243) 2025-03-25 15:14:10 -04:00
Evan Downing d8820285a4 Change build id to task id (#245) 2025-03-25 10:25:30 -04:00
Evan Downing 2240cdb617 Speedup graphml file creation (#200)
* broken paralell build of nodes

* fixes

* cleanup

* Add encoding task id

---------

Co-authored-by: Ian Smith <ian.smith@trailofbits.com>
2025-03-25 10:02:16 -04:00
Evan Downing c47f27debe Speedup graphdb (#193)
* Add bulk loading and update schema

* Remove unnecessary config options for speedup

* Uncomment unit test. Update integration test.

* Fix permission issue with removing temp folder

* Update file paths

* Changing graphdb directory

* Update integration config
2025-03-25 09:38:47 -04:00
Evan Downing 4a6c4baf07 Retrieve types using codequery (#201)
* Add retrieving types from treesitter

* Update preprocessor types. Update fuzz-tooling directory layout
2025-03-13 09:31:12 -04:00
Riccardo Schirone b44c21dd6e program-model: fix task-id property and improve tests (#174)
* program-model: fix task-id property and improve tests

* Skipping libpng integration test until kythe is ready

---------

Co-authored-by: Evan Downing <2077950+evandowning@users.noreply.github.com>
2025-03-12 15:20:39 -04:00
Riccardo Schirone c92e6c6a6f program-model: mount local source directory when indexing (#168)
* common: add workdir_from_dockerfile API in ChallengeTask

* program-model: mount local source directory when indexing

* program-model: configure graphdb_url from settings
2025-03-10 14:23:56 +01:00
Riccardo Schirone e0b898a6b4 program-model: introduce codequery (#178)
* program-model/tree-sitter: refactor classes

* program-model/tree-sitter: change how paths are considered

This allows to parse also code in oss-fuzz (e.g. injected fuzz targets
not in upstream project)

* common: change name format for challenge commit

* program-model/codequery: make it persistent and nice

* program-model: disable janus & co for now

* patcher: make use of codequery instead of just tree-sitter

* program-model: add integration tests for codequery

* program-model/codequery: add support for different languages

* patcher: install codequery in patcher's container

* ci: fix codequery install

* program-model: ensure cqdb storage in /crs_scratch

* program-model: ensure the cqdb storage is available only at the end
2025-03-10 14:11:07 +01:00
Henrik Brodin 0a7bac450e Henrik/indexing integration (#130)
Integration of the indexer service
2025-03-05 13:56:31 +01:00
2over12 a36fa80288 Switch to getting kzips using compdb fragments (#120)
* swtich to frags

* or true

* format

* add task id
2025-03-04 15:43:06 -05:00
Evan Downing d3f3cd3765 Add graphdb indexing (#114)
* Add indexes for fields

* Add indexing schema
2025-03-04 10:51:53 -05:00
Evan Downing e42b60e820 Add task id to graph (#94)
* Add task id field to nodes and edges

* Fix properties key
2025-03-04 10:51:35 -05:00
Evan Downing 0bfdfa2d7d Add get_function_body unit test (#96)
* Add get_function_body unit test

* Test passes

* Add janusgraph service

* Update tests with healthcheck
2025-03-04 08:09:25 -05:00
Evan Downing 2e5191b513 Build kythe into its own container (#104)
* Remove environment variables

* Add kythe release from base image

* Copy file from remote container

* Change kythe to remote

* x

* Fix dockerfile
2025-03-04 07:45:23 -05:00
Henrik Brodin 65dd397115 Store task-id in task-meta file (#123)
Store task-id in task-meta file
2025-03-04 13:00:13 +01:00
Evan Downing de724332ef Add try catches to program_model.py (#113) 2025-03-03 11:35:06 -05:00
Evan Downing f4446543ae Get function body by name (#73)
* Add skeleton for api call

* Modify encoding and decoding nodes/edges

* Add change encoding from base64 to hex

* reformat

* Target libpng function in example

* remove old project name

* cli fixes

* Add get_function via byte offsets

* refactor

---------

Co-authored-by: Ian Smith <ian.smith@trailofbits.com>
2025-02-25 10:21:08 -05:00
Riccardo Schirone 2d852a0881 program-model: add tree-sitter parsing for fallbacks (#90) 2025-02-24 09:58:21 +01:00
Evan Downing f6067005e6 Download Kythe automatically (#84)
* Add passing gh token

* Change docker to automatically download kythe release

* Update workflow
2025-02-21 13:33:38 -05:00
Riccardo Schirone 5b1ce401c2 Serve loop (#78)
* patcher: do not use package_name anymore
* patcher: fix tests
2025-02-20 14:26:42 -05:00
Henrik Brodin fe07435b77 Create the TaskMeta structure in the task directory (#79)
* Create the TaskMeta structure in the task directory

it contains information about project name and focus.

* lint

* Address review

* Restore name property
2025-02-20 03:48:23 +01:00
Evan Downing c6892939dc Updates program model directory (#66)
* Modify protobuf

* Update directory parameters for program model

* Update directory calls and build image call

* Always apply diff

* Always apply diff
2025-02-19 15:59:40 -05:00
Evan Downing 90d40ba4d1 Make graphdb storage persistent (#58)
* Add graphdb storage volume

* correct logging

* update volume
2025-02-18 18:02:00 -05:00
2over12 d4c9f74294 Enforce python >=3.10 and use ignore_cleanup_errors in TemporaryDirectory (#2)
* Enforce python >=3.10 and use ignore_cleanup_errors in TemporaryDirectory

* have an upper bound on requires-python

* update uv.locks

---------

Co-authored-by: Riccardo Schirone <riccardo.schirone@trailofbits.com>
2025-02-18 17:56:28 -05:00
Evan Downing 7b15a06214 change root of docker (#59) 2025-02-18 17:06:39 -05:00
Evan Downing af0ceb9e46 Program Model API (#25)
* Add doc skeletons

* Add API skeleton example

* Move graphdb url in settings
2025-02-18 17:06:15 -05:00
2over12 ea2fd093d5 Program model integrate (#13)
* Add documentation

* Add janusgraph custom configuration. Simply graph creation.

* Add properties to edges. Index on uri property.

* reformat

* Create libpng graph in a more reasonable amount of time.

* Update README.md

* Wrap indexing into function

* updating...

* update proto

* fix command

* Update settings for serve and processing indexing

* Add dockerfile to docker container

* Add trigger script

* Fix error with loading graphml file

* reformat

* Remove resource constraints for program model

* Change permissions of temp dir

* Update readme

* Update dockerignore

---------

Co-authored-by: Evan Downing <2077950+evandowning@users.noreply.github.com>
2025-02-18 11:59:16 -05:00
Riccardo Schirone 9885795582 program-model: add empty test to make pytest happy 2025-02-05 10:23:56 +01:00
2over12 c26a3e223d Add program model demo (#17)
* add program model uv

* add tool to common

* remove oss fuzz tool

* add docker

* implement build base image

* fix ret check

* add builder for subimage

* remove old tool

* hack to avoid dupped kzip files

* fixes... need to work out LD hack

* fix find

* justfile for building kythe

* update dockerfile

* add upload tool

* update protobuf

* move into compose

* fix merge invocation

* fix up invoc

* bump kythe

* reformat

* format

* format

* remove unused

* add lint and change version

* fixes

* exclude

* fix formatting:

* dev deps

* add dep

* exclude gen

* fix
2025-01-26 13:09:37 -05:00