Implement compression of objects before they are saved into the storage
provider. Each container type can specify which algorithm to use
(currently `none` or `zstd`) and the compression level.
Move around the code pertaining to the notification broker, making the
various roles a bit clearer. This also de-couple the broker logic from
the local revng daemon.
Switch out from using uvicorn to hypercorn for the pypeline daemon, make
the facilities create hypercorn commands in stand-alone functions, as
they will be re-used for other servers later on.
Fix the `revng2 project init` command to work with all
`StorageProvider`s, not just the local one. It still specializes for the
local one to allow creating the model file.
Expand the `ProjectMetadata` dataclass to include additional timestamps
and the hash of the pipeline description. Change all the
`StorageProvider`s so that all the new data is saved and returned.
Before this commit the pipeline description suffered from some
non-determinism that led to some values changing between runs of
`revng`. Add extra sorting so to eliminate the non-determinism.
Consolidate the saving of object dependencies and saving the actual
objects' data into a single function called `add_objects`. The previous
methods `add_dependencies`, `add_custom_invalidation_data` and `put`
have been removed.
Due to the drifting differences, split `ScheduledTask` into
`PipeScheduledTask` and `SavepointScheduledTask`, which allows each to
specialize on the task it has to accomplish.
Make the issuing of notification completely out of band with respect to
the `Pipeline`. Now the `StorageProvider` is responsible for providing
notifications to clients. Since some providers are local-only, there is
a `LOCAL_QUEUE` which allows sensing notifications through a local
queue, re-using the local revng daemon.
Split the `invalidate` method in two phases: in `invalidateCheck` it
just checks if the model diff warrants the execution of the actual
`invalidate` method, which requires fetching the custom invalidation
data from storage. The actual `invalidation` method remains the same.
This should make invalidation faster for storage providers that store
the invalidation remotely.
The `PypeGroup` click group did not provide any real benefit:
* It added extra help about pipebox arguments, but text that was not
correct from where the help was coming from (the group does not parse
the arguments). The subcommands still reports it though.
* It forbid using non-PypeGroup and non-PypeCommand commands in the
command tree, but in the long run this is desirable so it should be
dropped
For these reasons it has been dropped.
When one of the tasks completes, it might be due to an exception, call
the `exception()` method so that the exception can actually be caught by
the try block.
The `--format` option relied on `click.Context.get_parameter_source` to
determine if the default value for the `--format` parameter was passed.
A bug in click caused this to behave "correctly", click 8.4.0 fixed the
bug. Change the logic by using `None` as the default value so that it
can be checked without using the parameter source.
Add two situations where the cache is pruned on startup, removing all
the cached objects from storage:
1. the version field does not match the current one
2. the model has been changed out of band
The second one is very important as it allows to use the revng2 CLI
while editing the model manually on disk.
Overhaul the pipeline configuration logic by collapsing all dynamic
configuration options, both for analyses and pipes into a single
dictionary. Change all the interfaces so that there is no longer
distinction between the configuration of an analysis and of pipes.
Expose these options to the command line via `--{name}-configuration`
options for each pipe/analysis that is applicable to the command-line
invocation.
Ignore `OSError`s in `LocalStorageProvider` when rebuilding the list of
files for a given hash, since those paths might have become inaccessible
in the meantime.
Click clamps the output of its help to 80 columns, this makes reading
the help on wider terminals quite annoying. Change the formatter class
used to allow better help output on wider terminals.
Add an additional piece of metadata that states which pipes are used to
compute a specific artifact. This can be used to derive which
configuration options influence the creation of an artifact.
Rework the artifact endpoint, making it similar to the CLI invocation.
It now works on a single artifact and can be made to return both `json`
and `tar` as formats.
Change the `Container.serialize` interface so that it is possible to
supply a list of objects that will be serialized instead of all the ones
in the container.
Add two additional fields to `Artifact`: `defined_locations` and
`preferred_artifacts`. These are pieces of metadata that allow
navigation between multiple PTML-enabled documents.
Add categories in the pipeline, allow an artifact to have a category
specified which allows it to be shown or hidden by default. Adapt the
CLI tools to hide artifacts of the category that don't
`show_by_default=True`.
Since most functions in `pipeline_parser.py` had a lot of redundancy in
the parameters passed, convert all of them to be part of a
`PipelineParser` class which stores all the variables.
Rework which information is transmitted in the pipeline metadata,
avoiding redundancy and moving some information there instead of
returning it every time a request is made.
Giacomo Vercesi