Commit Graph

215 Commits

Author SHA1 Message Date
Giacomo Vercesi e2504dd6f2 Implement storage credentials update
Add the functionality to rev.ng's infrastructure to allow to dynamically
update the credentials of the underlying storage provider at runtime.
2023-09-14 17:32:08 +02:00
Giacomo Vercesi f710af9f2f Cleanup revng.daemon graphql API
Drop autogenerated queries from the GraphQL API and switch `revng.api`
to use `PipelineDescriptor`.
2023-09-14 15:44:33 +02:00
Giacomo Vercesi 2a239ddf47 Introduce PipelineDescription
Add the PipelineDescription object that describes the structure of a
pipeline. This will be used by all the downstream users instead of
pipeline accessor methods via PipelineC.
2023-09-14 15:44:33 +02:00
Giacomo Vercesi 90ea42b695 model/_util.py: move into tupletree.py
Move the contents of `model/_util.py` into `tupletree.py` as it's
model-agnostic and used by all generated Python tuple-tree wrappers.
2023-09-14 15:44:32 +02:00
Giacomo Vercesi e43e720a35 revng daemon: allow setting exposed CORS headers
Add the enviroment variable `REVNG_EXPOSE_HEADERS` for `revng daemon` to
allow customizing the headers exposed via CORS.
2023-09-06 16:12:43 +02:00
Giacomo Vercesi ca82e47f79 FunctionStringMap: switch to using GzipTarFile
Change the serialization and deserialzation functions of
`FunctionStringMap` to use the GzipTarFile. Also change che filename
of the underlying containers accordingly.
2023-09-06 15:23:43 +02:00
Giacomo Vercesi 93d3a6238d revng: introduce S3 support
Add the capability for all revng tooling to run with an S3-backed
workdir.
2023-09-06 15:23:43 +02:00
Giacomo Vercesi a84ebd0e51 revng.daemon: trim Eventmanager's functionality
Drop most of the functionality provided by EventManager. Add the
functionality that allows saving the working directory after a period of
inactivity.
2023-09-06 15:23:43 +02:00
Giacomo Vercesi e719b3b7e5 Add components to revng cli driver
This commit introduces the concept of `Component` to a pipeline step.
This, in turn, can be used by clients to figure out which artifacts are
produced by which revng component.
Additionally, this commit overhauls the `revng daemon-self-test`
command, renaming it to `revng graphql` and adding extra flexibility.
2023-08-23 17:45:36 +02:00
Giacomo Vercesi 9fc9ca0d1e revng.cli: centralize collect_pipelines
Move the logic that collects all the pipeline files to a cental
implementation.
2023-08-23 17:45:36 +02:00
Giacomo Vercesi 13ba286a9a Drop hard-coded analyses list
Remove any use of hard-coded analyses lists, relying on user interaction
(e.g. command-line arguments) to have it instead.
2023-08-23 17:45:36 +02:00
Alessandro Di Federico 247879f7fc model::Type::ID: switch to progressive IDs
This commit switches `model::Type::ID` from being a GUID to be a
progressive number, in order to make things easier for humans.

On top of this, this commit introduces the following changes:

* TypeCopier: import all the necessary PrimitiveTypes and improve
  handling of CustomName.
* Move Kind as the last field of the key of each TupleTree type used in
  an `UpcastablePointer`.
* Update the ground truth of tests to ignore the `CustomName` in favor
  of focusing on `OriginalName`.
* Increase adoption of `model::Binary::makeType`, equivalent to
  `Binary.recordNewType(makeType<model::*Type>())`.
2023-08-23 16:14:04 +02:00
Alessandro Di Federico fe18317bb7 model compare: introduce $ for exact size match
This commit introduces a change in `revng model compare` so that, for
dictionary entries whose keys start with `$`, it is ensured that the
length of the associated value (be it a list or a dictionary) is the
same.
2023-08-23 16:14:04 +02:00
Giacomo Vercesi 7e96bb2189 revng.api: avoid NULL dereferencing in extract
The call to `rp_container_extract_one` can return a null pointer, add a
check that allows to avoid dereferencing it if this happens.
2023-07-31 16:05:51 +02:00
Giacomo Vercesi 3a006422d3 typescript/python model wrappers: fix structure
Change the way the python and typescript model wrappers handle
attributes in struct fields. Now all fields are populated and none of
the use `None`.
2023-07-21 09:34:00 +02:00
Giacomo Vercesi 0ec2fdd175 revng.daemon: prevent shutdown crash
Change the behavior of `revng daemon` to avoid a crash at shutdown
caused by `rp_shutdown` not being called due to the pointer of
`rp_manager` not being freed.
2023-07-21 09:34:00 +02:00
Alessandro Di Federico 42abf64c62 revng.cli: ensure proper exit code propagation 2023-06-23 17:55:41 +02:00
Ivan Krysak 378f6e76f7 daemon: fix uploading binaries from html
The html view daemon sends when connected directly was not correctly
migrated to support analysis lists.
2023-05-25 06:25:07 +00:00
Giacomo Vercesi 7f73e2bfde revng.daemon: drop REVNG_PROJECT_ID
Remove the logic for computing a manager's workdir using the
REVNG_PROJECT_ID environment variable, as it was unused. The
REVNG_DATA_DIR remains to allow persistence across restarts.
2023-05-24 14:59:16 +02:00
Giacomo Vercesi 5d582a2eec fetch-debuginfo: improve cache folder handling
Rework how the cache folder for debug information is handled. This
commit introduces an additional environment variable, `REVNG_CACHE_DIR`
that allows overriding where semi-disposable files will be stored.
2023-05-24 11:09:04 +02:00
Giacomo Vercesi 2b181e627c fetch_debuginfo/common: improve download_file
Rework the code of `download_file` to leverage requests auto-decoding
option to auto-decompress the response body if needed.
2023-05-24 11:09:04 +02:00
Giacomo Vercesi 12186ff3a2 scripts/revng: re-implement path fix
Re-add the code that allows the `revng` script to be also invoked from
the build directory.
2023-05-11 14:24:00 +02:00
Giacomo Vercesi 31cd2b53f9 rcc: add codespell
Add codespell to the list of checks performed by
revng-check-conventions. This should reduce the amount of typos present
in the revng codebase.
2023-05-11 10:05:32 +02:00
Giacomo Vercesi a13c3e5ac2 revng: fix executable/shebang in scripts
Some files had discordant executable bit presence and shebang presence.
This commit fixes these occurrences by adding removing the required
feautures where needed.
2023-05-11 10:02:46 +02:00
Giacomo Vercesi 9d5b6d1f34 revng cli: migrate legacy scripts to _commands
Move the remaining commands in the `scripts` to be used by the `revng`
command line driver.
2023-05-11 10:02:46 +02:00
Giacomo Vercesi fdd07f640a revng cli: drop old code
Remove code for the `revng` command line entrypoint that was needed
before python became an orchestra component.
2023-05-11 10:02:46 +02:00
Giacomo Vercesi 471b9c8396 rcc: fix issues with mypy>=1.0
Add additional config options to the `revng-check-conventions` config to
make mypy work with the revng codebase (mainly related to untyped
functions now causing a warning in mypy).
2023-05-11 10:02:46 +02:00
Giacomo Vercesi 472710c100 daemon-self-test: Improve usability
Allow debugging commands (e.g. `--gdb`) to be passed through to the
invoked daemon when running `revng daemon-self-test`.
2023-05-09 17:27:00 +02:00
Giacomo Vercesi 41662b0a43 revng.support: Rework process invocation
Refactor the `run` and `try_run` functions in 4 separate functions:
* `try_run`: Run the process and return the exit code
* `run`: Run the process and assert the exit code was 0
* `exec_run`: exec the process, stopping execution of the python
              interpreter
* `popen`: Wrapper around `subprocess.Popen`, allows the most
           flexibility
2023-05-09 17:27:00 +02:00
Giacomo Vercesi 6ab8099531 revng daemon: disable CORS without --production
Set the `REVNG_ALLOWED_ORIGINS` to `*` when not in production mode. This
should avoid cors issues while developing.
2023-05-09 17:27:00 +02:00
Giacomo Vercesi a2b7d0e0bf PipelineC: add tracing
Add support for tracing onto the PipelineC. This is done by:
1. Creating wrapper functions for each PipelineC function with the
   script in `scripts/PipelineC_add_tracing.py`. These will call a
   special function called `wrap` which will ultimately call a method
   with a `_` prepended to the name
2. Conversion of all PipelineC methods in `PipelineC.cpp` to `static`
   and their rename with a `_` in front, in order for them to work with
   the wrapper function in (1)
3. Generation of 2 additional include files, one for types and one for
   functions, to be used by users of tracing files in order to have
   introspection.

These steps allow the creation of a trace file with the use of the
`REVNG_C_API_TRACE_PATH` environment variable. The traces can then be
used in conjunction with the `revng trace run` and `revng trace
inspect` commands.
2023-04-20 14:43:13 +02:00
Giacomo Vercesi 8d89c6a318 PipelineC: rp_initialize: only use argc and argv
Switch from specifying `libraries` and `pipelines` in `rp_initialize`
and `rp_manager_create` to the use of command-line options that are to
be passed via `argc` and `argv` in `rp_initialize`.
2023-04-20 14:43:12 +02:00
Giacomo Vercesi 3bfdb95d92 revng.api: Improve Manager.run_analyses_list
Fix the case where the return of `rp_manager_run_analyses_list` returns
nullptr.
2023-04-20 14:43:12 +02:00
Giacomo Vercesi 07be2c1cae Allow the use of valgrind with python
This commit does 2 main things:
* Adds PYTHONMALLOC=1 environment variable when `--valgrind` is used
  to force python to use normal malloc instead of the custom allocator
  python normally uses that's not friendly with valgrind
* Add a suppression file to remove a couple of false-positives that
  happen consistently and generate useless noise
2023-04-20 14:43:12 +02:00
Giacomo Vercesi 4400924312 revng.cli: allow non-ELFs to be run with gdb
If needed, wrap the execution of scripts in `exec` to allow running
them with gdb. This is especially useful for running python scripts
that use cffi and being able to inspect the C/C++ side.
2023-04-20 14:43:11 +02:00
Giacomo Vercesi 59175e9925 daemon-self-test: do not fail() twice
There was a codepath in `revng-check-conventions` that would cause
`fail(e)` to be called twice. This would cause an exception on the
second invocation due to the undelying process being already terminated.
2023-04-20 14:43:11 +02:00
Giacomo Vercesi 87543f835a revng.cli._commands: drop shebangs
Remove leftover shebangs from python files in `revng.cli._commands` as
those files are no longer executable on their own
2023-04-20 14:43:11 +02:00
Alessandro Di Federico 416c59c43b support.py: rework get_command
We introduce two paths where we can hard code search prefixes
(`/additional-search-prefixes`) and additional binaries paths
(`/share/revng/additional-bin-paths`).

This enables us to add to `additional-bin-paths` the configure-time
specified path of the preferred LLVM installation, instead of relying on
`PATH` to look up, e.g., `opt` at run-time.
2023-04-13 18:11:48 +02:00
Alessandro Di Federico 1fa34fff9e revng opt: force legacy pass manager 2023-04-08 08:42:24 +02:00
Alessandro Di Federico 79bbd8b260 commands_registry.py: drop dict comprehension 2023-04-08 08:42:23 +02:00
Kacper Kołodziej d281d1fa05 Add --binary argument to llvm_pipeline.py 2023-04-04 09:30:08 +02:00
Giacomo Vercesi 2e96e09f63 Drop analyze-all
Drop all the uses of analyzeAll and swap its use with the invocation of
the appropriate analysesList
2023-03-22 17:50:02 +01:00
Giacomo Vercesi af237074f3 Implement AnalysesList in Python API & GraphQL
Implement the new functionality of AnalysesList on the python's side of
rev.ng and the GraphQL schema both in static and autogenerated form
2023-03-22 17:31:46 +01:00
Giacomo Vercesi 23286dc58f GraphQL: expose analysis options
Add an additional parameter that allows specifying options for analyses
as a serialized json string
2023-03-22 17:31:46 +01:00
Giacomo Vercesi 1f097bb46a revng.api: Fix leftover snake_case in as_dict
This commit fixes the name of the fields returned by the `as_dict`
function that were left in snake_case when the field in the schema was
camelCase
2023-03-22 17:31:46 +01:00
Ivan Krysak 3687869795 Model: make AddPrimitiveTypes a pure analysis 2023-03-15 10:19:01 +01:00
Giacomo Vercesi 15da250710 api/daemon: fix single analysis handling
* Let GraphQL accept zero or more parameters for analyses
* Handle the general case of analyses using zero or more containers with
  a variable ammount of targets
2023-03-15 10:19:01 +01:00
Ivan Krysak 0b1aaeaede Minor improvements 2023-03-15 10:19:01 +01:00
Giacomo Vercesi 7751143aad revng daemon: add timeout-keep-alive option
Add the `--timeout-keep-alive` option for all invocations of `revng
daemon`. This avoids some edge cases where connections are abruply
closed.
2023-03-14 20:50:28 +01:00
Giacomo Vercesi 2b4cfafe55 revng.api: Fix file upload/buffer reading
This commit fixes file output `revng.api` in two ways:
* The buffer's size is properly reported in `PipelineC.cpp`
* The function `deserialize_container` in `revng.api.manager` no longer
  appends a `\0` to `content`, while avoiding a redundant copy
2023-02-16 15:11:00 +01:00