Commit Graph

403 Commits

Author SHA1 Message Date
Giacomo Vercesi d42cd5aeab pypeline: allow arbitrary config objects
Allow the `configuration` field of a pipe in a pipeline yaml file to be
of any type. When constructing the pipe object, the object will be
re-serialized via `yaml.dump` before being passed to the constructor.
2025-11-17 10:04:12 +01:00
Giacomo Vercesi c0e35d080e ContainerDeclaration: enable order
`ContainerDeclaration` needs to be orderable (e.g. have `__lt__`) to
allow `deterministic_sort_key` to work.
2025-11-17 10:04:12 +01:00
Giacomo Vercesi 824ec6194a PipelineNode: improve __repr__ function 2025-11-17 10:04:12 +01:00
Giacomo Vercesi d90f6ff0f5 revng2: use cache_directory from support 2025-11-17 10:04:12 +01:00
Giacomo Vercesi e38b55db64 pype project artifact: fix typo in code 2025-11-17 10:04:12 +01:00
Giacomo Vercesi ba4004691d pype project artifact: fix asking all objects
The `objects` argument was missing `required=False` which meant that all
invocations needed to specify the objects to request.
2025-11-17 10:04:12 +01:00
Giacomo Vercesi 32fd86d33a Pipe: drop name argument in constructor
Having a pipe's name different from its class name is useless, drop the
feature entirely.
2025-11-17 10:04:12 +01:00
Tommaso Fontana b38832e251 Pypeline: move epoch to storage_provider
Move the epoch inside the storage_provider, this allows us to make the
storage_provider_factory to return an async context manager.
This way we can simplify the locking for the daemon, but forces the
CLI project commands to deal with async code.
2025-11-11 17:15:28 +01:00
Tommaso Fontana 8ae70d85d4 Pypeline: replace logging with custom log
Now the pype command has a `--verbose` argument that to enable debug
logging in the pypelien code.
Adapted the codebase to use this new logging format but replacing
`logging` with `revng.pypeline.utils.logger`, this is done in
preparation of debug-log.
2025-11-11 17:15:28 +01:00
Tommaso Fontana 036411b441 Pypeline: add pipebox initialize and -- args
The pipebox now has to have an initialize function that is called with
all arguments passed to the CLI after --.
2025-11-11 17:15:28 +01:00
Tommaso Fontana a7f77b1704 Pypeline: propagate invalidated artifacts objects
Changed the API of storage_provider to treturn the invalidated object
ids and propagated this info to the CLI and daemon.
2025-11-11 17:15:28 +01:00
Tommaso Fontana 0ad8e8c849 Pypeline: add pypeline.daemon
Add implementation of the generic pype daemon.
It's implementation is split into `daemon.py` where we implement an
http framework agnostic interface, and `app.py` that uses it to serve
them using starlette.
2025-11-11 17:15:28 +01:00
Tommaso Fontana 71fad8b361 Pypeline: replace load_container with from_file 2025-11-11 17:15:28 +01:00
Tommaso Fontana 639593164a Pypeline: add autocomplete and refactor CLI
Now pype has an `autocomplete` command usable to enable autocompletion
and revng2 now is based on `pype`, modifying its defalults and
injecting new commands.
2025-11-11 17:15:27 +01:00
Tommaso Fontana 2daa9f938e Pypeline: add mime_type and model_name to Model
Now the model has a mime_type that we expect to return to the UI, and
model_name so the name of the model on disk can be impl-specific.
Moreover, now Model and Container use the same logic to detect if
a mimetype is textual.
2025-11-11 17:15:27 +01:00
Tommaso Fontana 652ffbb0f7 Pypeline: remove abbreviations, normalize quotes 2025-11-11 17:15:27 +01:00
Alessandro Di Federico fd216ef374 daemon: improve logging 2025-10-28 11:43:22 +01:00
Giacomo Vercesi 6abba05d7e Introduce ImportFiles pipe
Introduce the `ImportFiles` pipe which leverages the newly-introduced
`FileStorage` to fetch the required files and store them in the
`BinariesContainer`.
2025-10-16 17:48:45 +02:00
Giacomo Vercesi b622fe8f19 Implement FileProvider interface
Implement the `FileProvider` interface, this allows pipes to request
files given a `FileRequest` (hash and optional size/name).
2025-10-16 17:48:45 +02:00
Giacomo Vercesi 40da49548b Add Binaries list to the model
Add to the model an entry detailing the list of input binaries under
`Binaries`. This will be referenced by `Segments` when needed.
2025-10-16 17:48:45 +02:00
Giacomo Vercesi f17aa20bd7 model.migrations: implicitly increase version
Instead of delegating to the migration the task of increasing the model
version, do it implicitly as part of the `migrate` function.
2025-10-16 16:08:05 +02:00
Giacomo Vercesi 145bd01279 generate-migrations: remove __init__
Remove the `__init__` function from the template used by the
`revng generate-migrations` command, so far it hasn't been used.
2025-10-16 16:08:05 +02:00
Giacomo Vercesi 0f7dc7047d pypeline.Container: fix signature of (de)serialize
Change the signature of `Container.serialize` and
`Container.deserialize` to use `Buffer` instead of `bytes`, as this
makes the functions compatible with the `revng::pypeline::Buffer`
classes.
2025-10-16 16:08:05 +02:00
Giacomo Vercesi d08c490993 pypeline::Model: expose children function
Expose the `revng::pypeline::Model.children` function to python, as it
was not previously exposed.
2025-10-16 12:51:55 +02:00
Giacomo Vercesi 18f0bd88f9 pypeline::Model: fix serialize
The `serialize` function of `revng::pypeline::Model` erroneously
returned a buffer instead of `bytes`, fix the function so that `bytes`
are returned instead.
2025-10-16 12:51:52 +02:00
Giacomo Vercesi f165f99392 pypeline.Model: make deserialize to static
Change the `Model.deserialize` to a static function, which will return
a new instance of the Model.
2025-10-16 10:30:01 +02:00
Giacomo Vercesi c022c51b12 run_pipe.py: normalize arguments with kwargs
When click calls the command handler functions, dashes are converted to
underscores to work with the variable name limitations of python. When
accessing kwargs, convert the argument name to the underscored version.
2025-10-16 10:30:01 +02:00
Giacomo Vercesi 5c0ef7789e pypeline.storage: return invalidated objects
When calling `StorageProvider.invalidate`, return a dictionary that
conveys to the caller which objects were deleted from storage as a
consequence of the paths supplied to the `invalidate` call.
2025-10-15 17:43:09 +02:00
Giacomo Vercesi 5ea2cbdeaf pypeline.storage: fix invalidation
Fix the invalidation logic in both `InMemoryStorageProvider` and
`Sqlite3StorageProvider`, as it did not account for propagating the
invalidation to both children objects and parent objects.
2025-10-15 17:43:09 +02:00
Giacomo Vercesi d92f4cdca3 ObjectID: introduce byte (de)serialization
Add the `to_bytes` and `from_bytes` to `ObjectID`. This allows
serializing an `ObjectID` instance to raw bytes with a specific binary
format.
2025-10-15 11:58:41 +02:00
Giacomo Vercesi 59a7e0716a pypeline.storage.sqlite3: fix model path column
The column stored read model paths was erroneously called
`model_path_hash`, while the entire model was stored instead of a hash.
Rename the column to just `model_path` to avoid any confusion.
2025-10-15 11:58:41 +02:00
Giacomo Vercesi a1a38cf66d pypeline/cli/utils.py: fix wrong Kind usage
`Kind` objects do not have the `__name__` attribute, use the
`deserialize` function instead.
2025-10-15 11:58:11 +02:00
Giacomo Vercesi 6bdc3c9179 fetch-debuginfo: fix parsing of PDB filename
The PDB filename is up to the first `\x00`, everything else after should
be discarded.
2025-10-15 11:58:11 +02:00
Ivan Krysak 00ea93c99e TTG: move metaschema.yml to share/revng/ 2025-10-13 18:33:10 +03:00
Ivan Krysak 04fdc2028a migrations: do not duplicate the template 2025-10-13 18:33:10 +03:00
Ivan Krysak e56f543b1a Split module.bc.zstd 2025-10-13 10:26:41 +03:00
Ivan Krysak ed53e759c9 Do not use -Y flag
Yaml model are now the _only_ way to store them, no point in still
specifying the old flag for it.
2025-10-13 10:26:41 +03:00
Giacomo Vercesi 6d1fbdca97 Implement infrastructure for PipeRuns
Add the needed infrastructures that allow ephemeral classes (`PipeRun`s)
to be wrapped as a regular pypeline `Pipe` class. These automatically
have their dependencies tracked and propagated to the pypeline.
2025-10-03 17:40:45 +02:00
Lauri Vasama 653d451e6c Remove undef name configuration support
Also adds a model migration removing
Configuration.Naming.UndefinedValuePrefix.
2025-09-29 18:05:55 +03:00
Giacomo Vercesi 688b9fe111 Introduce compatibility with revng-pypeline
Add the necessary machinery to allow `Pipe`s, `Analysis`es,
`Container`s, `Model` and `ObjectID` to be implemented in C++ and used
by Python.
2025-09-17 14:02:40 +02:00
Tommaso Fontana 80eb02c207 Pypeline: merge pypeline 2025-09-10 12:05:15 +02:00
Tommaso Fontana 2778458539 Prepare CMakeLists for pypeline merge
Readme and pyproject for revng and revng_internal to the projects
folders.
Updated CMakeLists.txt so we can specify the README and pyproject
paths.
2025-09-10 12:05:15 +02:00
Tommaso Fontana 680f5cacf2 Apply revng check-conventions --force-format 2025-09-10 12:05:14 +02:00
Giacomo Vercesi 1a515afe07 CLIProject: fix _load_model
Fix the `_load_model` method of `CLIProject`. Before this commit it
erroneously did not call `_set_model` on all codepaths, resulting in
cases where `self.model` was incorrectly missing the `_project` field.
2025-09-08 17:36:12 +02:00
Giacomo Vercesi 62421eeaa2 fetch debuginfo: switch default servers
Switch the default servers used by `fetch debuginfo` to revng-managed
ones, as especially the elfutils.org once has been spotty in the past.
2025-08-08 17:52:57 +02:00
Giacomo Vercesi 9f63c7c476 revng daemon: handle PipelineManager no init
Handle the case in `revng daemon` where the `PipelineManager` fails to
init. In this scenario the server responds to all requests with a 500
(with the exception of `/` and `/status`, which return 200).
2025-07-30 15:43:02 +02:00
Giacomo Vercesi 140a49d088 revng daemon: handle ClientDisconnect exceptions
Gracefully handle the occurrence of a `ClientDisconnect` exception while
processing requests in `revng daemon`.
2025-07-30 14:54:17 +02:00
Giacomo Vercesi 615b2a69f8 synchronizer: avoid using multipart
Change the `TransferConfig` used by `S3Synchronizer._upload_file` to
effectively never use multipart upload.
2025-07-28 12:45:11 +02:00
Giacomo Vercesi f027129399 revng daemon: increase logging
Add additional logs to communicate when an upload happens, when a
produce is run and when an analysis is run.
2025-07-25 16:50:19 +02:00
Ivan Krysak 461c49df25 Model: bump version to 5 and introduce migration 2025-07-23 11:38:52 +02:00