140 Commits

Author SHA1 Message Date
Riccardo Schirone 37dad8a2dd patcher: improvements to handle auto-generated files
* program-model: index also .in files and others

* patcher: improve logic to better handle generated files

* patcher: update language-identifier binary

* do not check=True
2025-06-23 11:49:40 +02:00
Evan Downing b497098585 Re-enable tests (#877)
* Re-enable tests

* Fix tests
2025-06-21 16:07:26 +02:00
Riccardo Schirone 20b9fd61e1 Use 'git apply' to apply the diff, instead of 'patch' (#893) 2025-06-19 17:57:12 +02:00
Riccardo Schirone 9853d133a2 common|program-model: always use the most accurate container view (#860) 2025-06-17 16:37:07 +02:00
Riccardo Schirone a4a846df22 Clean tasks (#844)
implement scratch-cleaner daemonset
2025-06-13 19:10:44 +02:00
Evan Downing 0f68d6795f Propagate log max line length (#813) 2025-06-11 09:27:30 -04:00
Evan Downing 6d6bdba8cc program-model: fix tests (#799) 2025-06-05 16:26:00 +02:00
Evan Downing 4279960e2d Update oss-fuzz branch (#788)
Co-authored-by: Michael D Brown <michael.brown@trailofbits.com>
2025-06-04 23:34:40 -04:00
Riccardo Schirone 2540e2357c common: provide unified_language property in ProjectYaml (#774)
* common: provide unified_language property in ProjectYaml

* common: use also cpp

* Applying fixup to missed location

* codequery: fix how unified_language is used

* program-model: fix filter_project_context for new Language enum

* program-model: fix tests

* fix program model tests

---------

Co-authored-by: Michael D. Brown <michael.brown@trailofbits.com>
2025-06-04 23:33:32 -04:00
Riccardo Schirone 776732d017 program-model: use module scoped CodeQuery (#665)
* program-model: use module scoped CodeQuery

* fix also sqlite tests

* fix antlr4 test

* wip

* Fix macros test

* appease linting gods

* Fix cleanup routine

---------

Co-authored-by: Henrik Brodin <90325907+hbrodin@users.noreply.github.com>
Co-authored-by: Evan Downing <2077950+evandowning@users.noreply.github.com>
Co-authored-by: Michael D. Brown <michael.brown@trailofbits.com>
2025-05-21 16:40:55 -04:00
Evan Downing 36eee39bc6 Add commons-compress (#647)
Co-authored-by: Michael D Brown <michael.brown@trailofbits.com>
2025-05-20 14:28:49 -04:00
Evan Downing a23452d473 Add freerdp (#643)
* Add freerdp

* linter fix after merge

---------

Co-authored-by: Michael D Brown <michael.brown@trailofbits.com>
2025-05-20 14:17:53 -04:00
Evan Downing aed41348ce Add dropbear (#645) 2025-05-20 10:26:24 -04:00
Evan Downing 0f02c16c2c Add sqlite (#644) 2025-05-20 10:25:53 -04:00
Evan Downing 9537e58ae6 Add libpng from unscored-2 (#641) 2025-05-20 06:50:23 -04:00
Evan Downing 2d32b44c51 Update libxml2 after unscored-2 (#639) 2025-05-19 21:21:08 -04:00
Boyan MILANOV bef12fa3f5 Program model: Java type resolution (#613)
* Add basic get_type case

* Add support for dotted expression

* Add tests for method return type resolution

* Add more callees tests

* Lint and format

* Remove residual debug prints

* Commit tree_sitter file too

* Fix test issues from merging main

---------

Co-authored-by: Boyan MILANOV <boyanmilanov@coder-boyanmilanov-aixcc-boyan.c.production-1-405717.internal>
2025-05-19 16:07:34 -04:00
Evan Downing 7c71f0f11e Add additional tests for each existing challenge (#620) 2025-05-19 13:16:57 -04:00
Riccardo Schirone bfdf87bee9 program-model: function_name in get_types could be None (#627)
* program-model: function_name in get_types could be None

* linter fix

---------

Co-authored-by: Michael D. Brown <michael.brown@trailofbits.com>
2025-05-16 09:21:03 -04:00
Riccardo Schirone 093c62098c program-model: fix span attributes types (#614) 2025-05-15 09:39:48 -04:00
Michael D Brown de7d880bb1 More telemetry improvements (#592)
* #518 make fuzzing spans cover full effort

* #518 Adding gen_ai model name attributes

* add default for model name

* #593 add telemetry to fuzzing infra bots not running it

* format before push

* fix format issue induced by merge

* another linter fix

* Satisfy linter with updated type hints

* fixing imports

* fix formatting issue I forgot to fix before commit :(

* store default_llm to retain model_name param

* Add some program-model telemetry

* Added rest of new program-model telemetry

* #518 add more telemetry to fuzzing infra

* Revert "Satisfy linter with updated type hints"

This reverts commit 4a82dda091baa5705d97c97a397e9cecc6a646d9.

* #518 Add corpus merge telemetry

* fix wrongly generated parameter names.

* common: use service.instance.id to differentiate CRS runs

* fix lint

---------

Co-authored-by: Riccardo Schirone <562321+ret2libc@users.noreply.github.com>
Co-authored-by: Riccardo Schirone <riccardo.schirone@trailofbits.com>
2025-05-15 08:52:12 -04:00
Evan Downing 9c2efa307f Modify get_caller/get_callee arguments (#546)
* Modify get_caller/get_callee arguments

* Fix file_path filter

* Fix caller/callee checking

* Optimize get_caller/get_callee (#605)

* Uncomment log4j tests

* Move all logging to DEBUG

* Fix caller/callee logic
2025-05-14 13:01:20 -04:00
Evan Downing 99faaca54c Fix macro function bodies (#596) 2025-05-14 09:17:35 -04:00
Riccardo Schirone d72887d630 deployment: push logs to signoz (#591)
* Save logs in SigNoz

* deployment: push logs to signoz

* fix removal

* gitignore node_data_storage

* deployment: print SERVICE_INSTANCE_ID

* ci: configure otel
2025-05-13 12:16:08 -04:00
Henrik Brodin 75f159b04a Henrik/indexer expiry check (#590)
* Check expiry before indexing
2025-05-13 11:17:21 +02:00
Evan Downing a42d6083ac Add macro support (#503)
* Add macro tests

* Modify tree-sitter to ifdef lines

* Create no_preproc function

---------

Co-authored-by: Michael D Brown <michael.brown@trailofbits.com>
2025-05-09 20:56:39 -04:00
Evan Downing 5c1b2486fc Move challenge fixtures to conftest. (#569) 2025-05-09 16:14:06 -04:00
Evan Downing 104f4e8080 Remove unused parameters (#544) 2025-05-06 09:09:48 -04:00
Evan Downing dd7c9c21e7 Locate functions in java stacktrace (#460)
* Uncomment skipped tests

* Codequery was already working for bouncycastle.

* Add support for copying src after build
2025-05-01 13:12:44 -04:00
Evan Downing 17a7e77f87 Fix mypy errors (#515) 2025-05-01 07:22:04 -04:00
Riccardo Schirone 262f92098a program-model: use OSS_FUZZ_CONTAINER_ORG for base-image-url (#494) 2025-04-30 15:13:28 +02:00
Boyan MILANOV f55466101f Callee deduplication for C and Java (#495)
* Add basic imports resolver

* Fixes and more tests

* Doc

* Enable C callee dedup in codequery

* Add two libxml2 tests

* Add test to log4j

* Add java callee dedup

* Use container src dir, fix zookeeper tests

* Add test workaround for log4j2

* Lint

---------

Co-authored-by: Boyan MILANOV <boyanmilanov@coder-boyanmilanov-aixcc-boyan.c.production-1-405717.internal>
2025-04-30 14:54:39 +03:00
Riccardo Schirone 6127ca8786 Wait for DinD to be ready (#490)
* deployment/charts: start dind first

* Wait for DinD to be up and running
2025-04-30 09:54:26 +02:00
Evan Downing b1a1c19d99 Modify fuzzy search (#484) 2025-04-29 15:37:25 -04:00
Evan Downing 558c0aeb97 Update Ubuntu version of program-model (#472)
* Update Ubuntu version to support newer version of CodeQuery

* program-model: install custom cscope and updated codequery

* patcher: fix vuln_id -> submission_index

* patcher: fix docker install

---------

Co-authored-by: Riccardo Schirone <riccardo.schirone@trailofbits.com>
2025-04-29 15:20:14 -04:00
Michael D Brown 83fe3eb8b9 Add CRS Action telemetry (#462)
Adds telemetry for all services including required and some extra attributes.
2025-04-29 14:48:37 -04:00
Evan Downing 2f4fabf792 Add more robust fuzzy search (#464)
* Add regression test for fuzzy search

* Add more robust fuzzy matching logic
2025-04-25 15:51:16 -04:00
Evan Downing d62d1fb4b8 Add option to pass just the function name (#463) 2025-04-25 15:24:00 -04:00
Evan Downing 2273c2406e Fix errors (#461)
* Fix errors

* Revert lru_cache change
2025-04-25 09:58:16 -04:00
Evan Downing 8377739753 Resolve mypy errors (#459) 2025-04-25 07:52:57 -04:00
Evan Downing ef323e20e9 Remove logger calls. Redirect subprocess run outputs (#458) 2025-04-25 07:49:57 -04:00
Evan Downing 93f55a274b Add support for inner classes (#443)
* Change cscope call to compress database and improve lookup speed. Update tests.

* Fix log4j2 source paths

* Add private version of cscope
2025-04-25 07:37:39 -04:00
Evan Downing 73381dbe0e Add support for log4j2 (#406) 2025-04-17 16:28:10 -04:00
Evan Downing a038b48330 Add support for bouncy castle (#405) 2025-04-17 16:13:16 -04:00
Evan Downing bf45c6ac44 Adding regression tests for graphql (#403) 2025-04-17 16:08:42 -04:00
Evan Downing 184df8b506 Add regression tests for checkstyle (#404) 2025-04-17 15:46:45 -04:00
Evan Downing d975edf0a9 Add support for commons-codec (#390)
* Add support for commons-codec

* Uncomment passing test

* Move common file. Modify zookeeper and commons_codec tests.
2025-04-17 15:33:22 -04:00
Boyan MILANOV d2bf90e974 [Program-model] C tests for HDF5 (#407)
* Refactor get_functions test

* Add some tests

* Callees tests

* Add type defs test

* Remove residual debug code

* Fix tests

* Fix workflow file

* Fix workflow file again

* Add file_path tests for type defs, implement path rebasing for type defs in codequery

* Fix fixtures in pytest

* Fix module scoped fixtures in tests

---------

Co-authored-by: Boyan MILANOV <boyanmilanov@coder-boyanmilanov-aixcc-boyan.c.production-1-405717.internal>
2025-04-17 19:37:43 +03:00
Riccardo Schirone 2475b8e507 Move to use in-container paths (e.g. /src/) (#395)
* disable kythe/graphdb in tests
* fix integration CI
2025-04-16 11:05:44 +02:00
Boyan MILANOV d68c7d6a96 Fixes in CodeQuery and TreeSitter APIs with tests (#399)
* Add callees tests

* Add more tests for finding types

* Remove debug statements
2025-04-16 10:34:32 +02:00