Before this commit the `invalidate` code did invalidate all the objects
but left all the rows in `dependencies` that were paired to the produced
objects except the invalidated paths.
In this commit the logic is changed to be done in 3 steps:
1. SELECT all the objects that need to be invalidated, write down their
rowid
2. Delete all the rows in `dependencies` that were produced in
conjunction to the objects from (1)
3. Remove the objects from the table, returning invalidation data
This guarantees that the database does not accumulate invalidation data
over time due to invalidations.
Change the `get_model` and `set_model` interface of `StorageProvider` so
that it is responsibility of the `StorageProvider` to
serialize/deserialize the model before returning to the caller.
This is in preparation to the model migration being implemented, since
it's now a responsibility of the storage provider to deserialize it it
can trivially re-save it to disk if it is migrated.
Overhaul the logic and method names involved in enabling and disabling
reference caching in `TupleTree<T>`. `TupleTreeReference<T, U>` now
lazily caches the target and will traverse the path only when needed.
Also expose and use these functions in the new pipeline, which should
provide some speedup when executing a `Schedule`.
The functions that take care of running an analysis/analysis list have a
lot of duplication, merge the two into a single function that takes care
of running both.
Fix the code that handles setting the `--format`, `--tar` and `--yaml`
command line options, which incorrectly wrote to the `format` kwargs
instead of the `container_format` one.
Instead of executing `chdir` when using the `-C` option, which is
fragile when other paths are involved, store the option in the click
context and propagate it to the required code paths that require knowing
what the base directory is.
Allow the storage provider to be initialized with `local://?inline`.
This will set the cache directory to `$PWD/.cache` instead of
`~/.cache/revng`. This is to simplify testing for pypeline.
Add the functionality the the pipeline infrastructure and CLI to run
individual pipe and analyses as subcommands instead of in-process. This
allow better debuggability of individual pipes.
In preparation for debugging individual pipes add an additional method
to the `Pipe` interface, `needed_files`, which allows the pipe to
statically report which files it will request ahead of time.
Convert the `RuntimeError`s emitted by the pipebox (e.g. when checking
preconditions or running analyses) into `PypelineError`s and have the
pypeline cli infrastructure handle them gracefully, without emitting
stacktraces.
Add infrastructure to pypeline that allows containers to be notified
when they are being used last, this allows two things:
* `Pipe`s eagerly clearing those containers once they are done reading
their contents
* `ScheduledTask`s clearing those out at the end of their execution in
case the pipe did not do it
This overall should improve memory usage as container no longer take up
memory if they are no longer used as part of a `Schedule`.
Inline the body of `PipelineNode.run` into `ScheduledTask.run`, making
`PipelineNode` a pure data structure. This also allows eliding all the
arguments since all of them as properties of `ScheduledTask`.
The function `Request.check` did not work properly due to
`Container.contains_all` performing the check with the operands swapped.
Fix the call and improve the body of `ObjectSet.issubset`.
Also improve the error message, highlighting the missing objects.
When setting the variable names for the analysis' configuration,
sanitize the analysis' name, as it might contain dashes and other
undesirable characters.
When running `invalidate`, use a temporary table to store the
invalidated model paths that need to be used. This saves to template
them into the query and allows them to be recycled later when the
`dependency` table needs to be cleaned up.
Now commands 'pype project artifact' and 'pype pipeline run_pipe' now
accepts --format to specify in which format to serialize the resulting
container, and have --tar and --yaml shortcuts. Moreover, these commands
and 'pipe pipeline run_analysis' can automatically figure out the format
of containers so the user won't have to specify them most of the time.
Now containers can be dump and loaded in both yaml and tar. Also the
container can automatically try to figure out the format of the
container. Moreover, now containers have to_bytes and from_bytes.
Optimize the `run` function to skip objects that have just been
serialized when deserializing objects for outgoing requests. For
example, previously if a schedule had for a savepoint:
```yaml
incoming: ["/binary"]
outgoing: ["/binary"]
```
then the `Savepoint.run` function would have first serialized the object
into storage and then deserialized it right after, instead with the new
function the object is only serialized.
Overload the `get` function of `Requests` so that, in addition to the
usual `Mapping` behavior, will return an empty `ObjectSet` if no default
is specified.
Improve the `schedule` in the `Pipeline` class by introducing two
optimizations:
* Prune all the `ContainerDeclarations` that are not actually used.
* Skip all tasks that have no outgoing elements that would have been
written by them.
Allow the `configuration` field of a pipe in a pipeline yaml file to be
of any type. When constructing the pipe object, the object will be
re-serialized via `yaml.dump` before being passed to the constructor.
Giacomo Vercesi