Propagate traits to sources/inputs

[jmilloy]

Some traits, such as the caching, need to be propagated to Compositor sources, Algorithm inputs, or the Interpolator source. Currently, nodes handle this individually and incompletely.

We will use a tag, such as sync or propagate, to mark which traits need to be propagated.

tl.bool(cache_ctrl).tag(propagate=True)

We discussed propagating these tagged traits at eval time if possible, ideally in Node.eval or in the various _eval methods if necessary. Note that Compositor sources are often instantiated as needed in different ways, so we need to make sure those cases are covered.

We also discussed specifying not just that a trait needs to be propagated, but where it needs to be propagated to for fine-grained control.

tl.bool(cache_ctrl).tag(propagate='sources')
tl.bool(cache_output).tag(propagate=['a', 'b'])

Traits should only propagate if it exists but has not been defined in the target node.

[jmilloy]

@mpu-creare This still bites me sometimes. I'm looking forward to fixing this. I'm okay with the propagate flag that we speced together, but I wonder what you think about gathering these traits into a single configuration environment that can be propagated. This would be a some (but not all) of the settings as an object that can be easily copied and modified and propagated. Each node would just have a single 'env' trait.

env = podpac.Environment(cache_ctrl=['ram'])
node = MyCompositor(env=env)
node.sources[0].env.cache_ctrl # ['ram']

# env traits can also be set like this
env = podpac.Environment()
env.multithreading = False

A context manager would be nice:

with podpac.Environment(cache_ctrl=['ram']) as env:
    node = MyNode(env=env)
    node.eval(coords)

Especially if the environment was automatically populated (not sure if this is feasible):

with podpac.Environment(cache_ctrl=['ram']):
    node = MyNode()
    node.eval(coords)

I'm not sure if this is feasible, either, but there is potential to evaluate the same node with different configuration environments:

node = MyCompositor()
node.eval(coords)
with podpac.Environment(cache_ctrl=['ram']):
    node.eval(coords)
with podpac.Environment(cache_ctrl=['disk']):
    node.eval(coords)

Keyword arguments for ease of use. These set up the environment under the hood:

node = MyNode(multithreading=True)

# same as
with podpac.Environment(mulithreading=True) as env:
    node1 = MyNode() 

If each eval can have it's own environment context, then the keyword arguments would work there as well:

node = MyNode()
node.eval(coords, multithreading=True)

# same as
node = MyNode()
with podpac.Environment(mulithreading=True):
    node.eval(coords)

Like settings, there would be a default, and in fact you could save other environments for later use.

env = podpac.Environment() # load default
env.cache_ctrl = ['ram', 's3']
env.aws_region_name = 'us-west-1'
...

# save
env.save('my-environment")

# load
env = podpac.Environment('my-environment')

Basically, I'm proposing replacing the Node cache_ctrl, cache_output, force_eval traits with an env trait, which holds additional evaluation settings, such as the AWS settings, request session, multithreading configs, DEBUG, etc. Some global settings would not be included in the evironment such as PODPAC_VERSION, DEFAULT_INTERPOLATION, DISK_CACHE_ENABLED, etc.

It would be easy to implement a simple environment to cover this issue specifically, and then enhance as needed.

[mpu-creare]

Yeah, that's a great proposal... but I'm not ready for this quite yet. I'm planning to do a release next week to support the GeoWATCH work. After that we can start adding additional features again.

[jmilloy]

Yeah, this would be potentially be in the 4.0 major release.