* Use an internal_patch_id instead of indices
This is the first step in being able to merge sets of PoVs and test
patches against all PoVs within a task.
* Discard redundant builds
* Initial PoV-merging strategy
Still not optimal in terms of SARIF-matching/bundling etc
* Appears to be working version of merging including bundle and sarif handling
* Make tests pass
* Update integration test steps
* Fixes and cleanup from review
* Removed additional request for patched builds
* Refactored some loops into find-style functions to simplify
* Inline small function used once
* Refactors for increased robustness and readability including additional testing
* SARIF matching - additional tests and refactor
* Add enumerate_task_submissions
* Refactor and simplify tests using a builder
Cleanup unused code
* Fix read_submissions to use CrashWithId
* Improvements based on review
* Cache final states of PoV reproduce (#909)
As these never change we can limit the load on redis by caching the
results.
* Merge SubmissionEntries based on patches (#910)
* Cache final states of PoV reproduce
As these never change we can limit the load on redis by caching the
results.
* Merge SubmissionEntries based on patches
If a PoV in another entry is mitigated by the current entry's patch,
merge the entries as athey should be considered the same
ChallengeVulnerability.
* Add positional argument
* Hold of submitting a patch while evaluating
Check each already submitted patch before submitting a new one for the
same task. If any of the already submitted patches mitigates any PoV in
the current SubmissionEntry - do not submit this. It will be merged
later on.
* Additional logging, truncate ids
* Only request patch if no submitted patch mitigates
Before we request a new patch, we check each of the already submitted
patches to see if any of them already mitigates the PoVs in the current
SubmissionEntry. If they do, this will be merged at a later stage.
* PR feedback
* Use an internal_patch_id instead of indices
This is the first step in being able to merge sets of PoVs and test
patches against all PoVs within a task.
* Discard redundant builds
* Compare relevant fields
* Initial orchestrator deduplication
This is the first phase handling of PoV duplicates. PoVs having
identical crashes are stopped at the fuzzer. Others migrate to the
orchestrator. In the orchestrator PoVs are compared to see if they are
'similar' to any other PoV. If so they are bundled and each are used to
validate patch quality before submitting.
This PR basically acts as a filter for only submitting a patch for any
PoV considered similar before the patch can be submitted the first time.
* Make sure duplicates are only considered for the same task
---------
Co-authored-by: Michael D Brown <michael.brown@trailofbits.com>
* Make builder-bot build patched versions
* orchestrator: test new build outputs
* address review
* builder-bot: use times_delivered logic in case of failures
* Cleanup of submission logic
Retries patches, and moves on to next patch if available
Handles the errored state
Prepares for SARIF submission (missing the SARIF matching)
Patches bundles to include SARIF
* remove whitespace
* Limit impact of exceptions
If an exception is raised while processing a vulnerability, we
would previuosly stop processing any vulnerabilities. This change
enables processing to continue for other vulnerabilities.
* Make build type an int enum
* Lint
* Fix protobuf
* Fix protobuf for linux
* Remove old enum and use the class from protobuf message
* Lint
* Fix tests
* Print build type name instead of int value in logs
* Try fix enum instanciation in scheduler
* Correctly get protobuf enum name for BuildType values using EnumTypeWrapper methods
* Fix import
---------
Co-authored-by: Boyan MILANOV <boyanmilanov@coder-boyanmilanov-aixcc-boyan.c.production-1-405717.internal>
* Calculate function coverage fraction
Note: function isn't tested yet
* Store function coverage in Redis
* Sample function off coverage map in seed-gen
* Conditionally update function coverage
* Try multiple definition lookup heuristics
* Log function probability and coverage fraction
* Fix calculate_function_probabilities
* Fix docstring
* Implements task cancelation
- Cancel one or all tasks.
- delete_task.sh script for invoking
- refactored to keep deleted tasks in a separate set, update tasks as
needed from registry instead of updating the stored tasks.
* Refactor cancellation, update use in vulnerabilities
* Cleanup cancellation
* Remove package_name from protomsg
* Fix test case
* fix tests
* PR feedback
* Create the TaskMeta structure in the task directory
it contains information about project name and focus.
* lint
* Address review
* Restore name property
* Implement actual build request creation
Removes the mock and adds functionality to parse the project yaml.
* Move the ProjectYaml to common
* lint
* Pass in the correct name
* pr feedback
* Tmpfix uvlock
* Add documentation
* Add janusgraph custom configuration. Simply graph creation.
* Add properties to edges. Index on uri property.
* reformat
* Create libpng graph in a more reasonable amount of time.
* Update README.md
* Wrap indexing into function
* updating...
* update proto
* fix command
* Update settings for serve and processing indexing
* Add dockerfile to docker container
* Add trigger script
* Fix error with loading graphml file
* reformat
* Remove resource constraints for program model
* Change permissions of temp dir
* Update readme
* Update dockerignore
---------
Co-authored-by: Evan Downing <2077950+evandowning@users.noreply.github.com>
* apply diff in builder bot
* fix dockerfile seed-gen
* format
* apply the diffs
* fix builds
* store if diff was applied
* Revert "format"
This reverts commit f07a806ebd98f16409f0c3676aefc50441562239.
* add stacktrace for consumers
* add set
* crash dedup
* update dedup to include info that prevents clash accross diff task
* update key
* fix flag
* format
* actually point to set object
* fix to be correct
The two protobufs fuzzer_msg.proto and orchestrator.proto has been
replaced with a single msg.proto. This is to make it easier to embedd
structures in other structures across protos. Just compoling the protos
work well, but python imports break. As there aren't that many
structures merging seamed like an easy fix.
Ties the DELETE task http endpoint to the registry and marks tasks as cancelled.
Further, it tracks the timeout for a task and marks expired as cancelled.
* orchestrator/scheduler: initial implementation
* orchestrator/scheduler: mock libpng
* orchestrator/scheduler: process each type of msg independently
* common: move protobufs to common to be used by all components
* common: check same-reader pending list and autoclaim
* common: add claim_item method
* common: refactor queue/group names into Enums
* common: add generic typing for reliablequeue
* common: make the pending items list check non-default
* common: use item_id directly in ack_item
Henrik Brodin