Implement compression of objects before they are saved into the storage
provider. Each container type can specify which algorithm to use
(currently `none` or `zstd`) and the compression level.
Consolidate the saving of object dependencies and saving the actual
objects' data into a single function called `add_objects`. The previous
methods `add_dependencies`, `add_custom_invalidation_data` and `put`
have been removed.
Split the `invalidate` method in two phases: in `invalidateCheck` it
just checks if the model diff warrants the execution of the actual
`invalidate` method, which requires fetching the custom invalidation
data from storage. The actual `invalidation` method remains the same.
This should make invalidation faster for storage providers that store
the invalidation remotely.
In preparation for debugging individual pipes add an additional method
to the `Pipe` interface, `needed_files`, which allows the pipe to
statically report which files it will request ahead of time.
The function `Request.check` did not work properly due to
`Container.contains_all` performing the check with the operands swapped.
Fix the call and improve the body of `ObjectSet.issubset`.
Also improve the error message, highlighting the missing objects.
Optimize the `run` function to skip objects that have just been
serialized when deserializing objects for outgoing requests. For
example, previously if a schedule had for a savepoint:
```yaml
incoming: ["/binary"]
outgoing: ["/binary"]
```
then the `Savepoint.run` function would have first serialized the object
into storage and then deserialized it right after, instead with the new
function the object is only serialized.
Overload the `get` function of `Requests` so that, in addition to the
usual `Mapping` behavior, will return an empty `ObjectSet` if no default
is specified.
Giacomo Vercesi