Update on lineapy.to_pipeline
#742
mingjerli
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
We are one step closer to removing the duplicated code execution in pipelines generated from
lineapy.to_pipeline.Last week, we have merged the following two pull requests PR-714 and PR-734.
In PR-714, we've analyzed the session graph to identify the dependencies between any two artifacts within the same session and extract common code to avoid executing multiple times.
In PR-734, we've extended the work of PR-714 from a single session to multiple sessions, so we can easily stitch artifacts in multiple notebooks into a single pipeline.
Note that, we no longer need to specify task dependencies within the same session (thanks to PR-714), but we still need to specify task dependencies between different sessions(but only need one task from each session).
We are currently rewriting the current
lineapy.to_pipelineusing the work from these two PRs, and users should not expect any API changes once we update the implementation.Beta Was this translation helpful? Give feedback.
All reactions