Replace pythonjob task by normal task, drop usage of pickle_node#35
Merged
jan-janssen merged 7 commits intopythonworkflow:mainfrom Apr 2, 2025
Merged
Replace pythonjob task by normal task, drop usage of pickle_node#35jan-janssen merged 7 commits intopythonworkflow:mainfrom
pickle_node#35jan-janssen merged 7 commits intopythonworkflow:mainfrom
Conversation
03ab7db to
7c4d9df
Compare
3 tasks
Collaborator
|
Thanks for the nice changes, @superstar54! (both here, as well as in WG itself) |
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In my workstation, it took ~7 mins to run the QE workflow using AiiDA-WorkGraph, while other engines only take ~ 1 mins. The main reason is that the
PythonJobtask was used to auto-serialize inputs/outputs and preserve data provenance. However,PythonJobis designed to run the Python function on a remote computer. It uploads inputs to a remote folder and retrieves outputs to the local computer. In this project, it's overkill. In the latest aiida-workgraph, the normal keep provenance too. So this PR drops the usage of the PythonJob task.Another reason is that, the
pickled_nodetask is created for each data node, in this PR, when loading the workflow from a json file, the normal task can also serialize the inputs/outputs in an automatic way, so thepickle_nodeis not needed anymore. When writing the workflow using AiiDA, we prepare the AiiDA data nodes as inputs, sopickle_nodeis not needed either.This PR reduces execution time from ~7 mins to ~ 1.5 mins.
This PR needs
aiida-workgraph>=0.5.1.