6 Commits

Author SHA1 Message Date
Giacomo Vercesi dfb9a36bb2 pypeline: implement compression
Implement compression of objects before they are saved into the storage
provider. Each container type can specify which algorithm to use
(currently `none` or `zstd`) and the compression level.
2026-06-17 10:06:35 +02:00
Giacomo Vercesi 7f7f9571a8 pypeline: implement container discarding
Add infrastructure to pypeline that allows containers to be notified
when they are being used last, this allows two things:
* `Pipe`s eagerly clearing those containers once they are done reading
  their contents
* `ScheduledTask`s clearing those out at the end of their execution in
  case the pipe did not do it
This overall should improve memory usage as container no longer take up
memory if they are no longer used as part of a `Schedule`.
2025-12-19 10:29:28 +01:00
Giacomo Vercesi e8f293271b BinariesContainer: page files to disk
Store the contents of the files deserialized by `BinariesContainer` to
disk via `TemporaryFile`. This also allows some pipes to avoid shuffling
data around since they needed a file on disk anyways.
2025-12-19 10:28:29 +01:00
Giacomo Vercesi 97b471c888 Add Lift to pypeline 2025-11-18 17:47:54 +01:00
Giacomo Vercesi 4f93c9e7ad Container: add mime type
Add the `MimeType` field to the C++ classes. Forward it to python via
nanobind.
2025-11-17 10:04:13 +01:00
Giacomo Vercesi 6abba05d7e Introduce ImportFiles pipe
Introduce the `ImportFiles` pipe which leverages the newly-introduced
`FileStorage` to fetch the required files and store them in the
`BinariesContainer`.
2025-10-16 17:48:45 +02:00