-
Notifications
You must be signed in to change notification settings - Fork 7
Description
In the current DaskResolver, we sometimes construct the task for calling an algorithm with the ConcreteAlgorithm as the first argument, or sometimes we use ph_apply(func, args, kwargs) (https://github.com/metagraph-dev/metagraph/blob/main/metagraph/core/plugin.py#L618-L628) as the first argument, effectively "currying" the keyword arguments since Dask doesn't have a direct way to represent them in the task.
The ambiguity between these two scenarios complicates #166, which needs to walk the graph and find compilable tasks. If we expand ph_apply() to a full-blown callable class (MetagraphTask) and always use it, even when only positional args are needed, we can streamline some of the compilation logic as well as have a place to stash metadata attributes about the task that Dask otherwise has nowhere to put.
In fact, I think this would eliminate the need for #180, as we could stash the resolver reference in the MetagraphTask and pass it down to the ConcreteAlgorithm when needed for compilation, or other use.
@jim22k: Thoughts? If this seems reasonable, I can do it as part of #166.