runtime asset Redirection with Layered Loading System

[Lypsolon]

this pr provides runtime and load time redirectoin for assets consumed via the usd resoler.

the idea is as follows.

• provide a RDF(redirection file)
• the rdf will say "fileA"->"pathB"
• the resolver will then first check for an rdf and use this data to early return and avoid other reselution systems.

Implementation

the rdfs are implemented as thread save classes using shared mutex for parallel read and single write.

they are stored in an defined format as a json file

they allow layering and the first file loaded will be the strongest allowing for composition.

there is a function getRdFile that allows you to get an RDF instance via an abs file path (the file path dose not need to be abselute but all internal systems use an abs file path and you need to fully match the file path to get the right instance)

using the system

Option A

the most straight forward option is to have a stage that has an rdf configured in its custom layer metada

#usda 1.0
(
    customLayerData = {
        token AyonUsdResolverRedirectionFile = "path/to/rdf.json"
    }
)

it is important that we use an TF token becuase other values do not allow for . in the path.
also internally we need to cast the value at the token so if its not an tftoken its gonna end up with UB.

the root usd will be loaded via _CreateDefaultContextForAsset and we will extract the rdf file via searching for the key AyonUsdResolverRedirectionFile.

• this allows us to store different things and extend the system in the future by adding other keys
• this also allows easy association between a given stage and its redirctoin file(the assumption is that most of the time you dont want to gloably redirect an asset but you want to do it for a given stage)

Option B (not working)

it is wanted that we can just create a stage and then add an redirdctoin file to the resolver context to allow for redirection. (we could then use reload or AR::notice to reload some paths)

this dose not work for the following reasons

• there is no way to get the curretnly active resolver context from a given stage and then modyfy it. (this is posible if you created the context and passed it at stage creation) (no GetPathResolverContext dose not return the right object and AR.GetResolver() dose only gives the parent class, also the connected context via GetCurrentContext dose not apply to the stage)

implementation and why

Recursive rdf files

rdf files allow for sublayers just like usd dose. this allows the follwing.

• modelingLayer overwrites assetA -> v2
• lighting layer now imports modeling layer, in this moment we want to inherent the redirection they probaly redirected for a reason. we can now overwrite if we want to.

getRdf func

this funciton caches rdf files like a singleton would do. its wraped in a funciton like this to avoid common problems with named singeltons, this also allows us to create rdf files and modyfy them without having to care for the in memory instances.

[Lypsolon]

after a talk with @BigRoy we came to the idea that every time the resolver context is create and dose not have an RDF connected we will create an RDF instance based on an uuid and then when we get hold of a class copy of the resolver conext we can get acess to set RDF via the uuid

[BigRoy]

Also wanted to drop this screenshot of Houdini on how to configure a /stage-wide resolver context:

Mostly because I keep forgetting myself where that button is :P

[BigRoy]

I'm looking to test this PR and have the resolver inside Houdini now. I'm just not sure which of the example test codes to use and how they apply to a workflow in Houdini.

Do note that with default settings on creating a single Load Asset LOP I get this Houdini console output:

randId rdfL7D1oA6SYU8Rq4ueKdXZw8dQyt8ha9OhB1kU158T6zQHIiSmksuaoNyG9FmoPCN0
redirectionFile: "C:\\Users\\User\\AppData\\Local\\Temp\\L7D1oA6SYU8Rq4ueKdXZw8dQyt8ha9OhB1kU158T6zQHIiSmksuaoNyG9FmoPCN0.json"
Warn: Cant find redirectionFile. Redirection File will be created
getLayerStack //Layer File was found in a higher Layer: "C:\\Users\\User\\AppData\\Local\\Temp\\L7D1oA6SYU8Rq4ueKdXZw8dQyt8ha9OhB1kU158T6zQHIiSmksuaoNyG9FmoPCN0.json"

randId rdfR16cGbsP7MpICSoIcY48f7wDE5fo2ABilhEqSLWJFIM8usoXmdHtZaYR4YZvM7TX
redirectionFile: "C:\\Users\\User\\AppData\\Local\\Temp\\R16cGbsP7MpICSoIcY48f7wDE5fo2ABilhEqSLWJFIM8usoXmdHtZaYR4YZvM7TX.json"
Warn: Cant find redirectionFile. Redirection File will be created
getLayerStack //Layer File was found in a higher Layer: "C:\\Users\\User\\AppData\\Local\\Temp\\R16cGbsP7MpICSoIcY48f7wDE5fo2ABilhEqSLWJFIM8usoXmdHtZaYR4YZvM7TX.json"

randId rdfND7amS1xUnQJoFY7pLWXORoZBs40QeR8wMULXK8Kvi9d7vjz0N0X8Pl4qlQ0kMnp
redirectionFile: "C:\\Users\\User\\AppData\\Local\\Temp\\ND7amS1xUnQJoFY7pLWXORoZBs40QeR8wMULXK8Kvi9d7vjz0N0X8Pl4qlQ0kMnp.json"
Warn: Cant find redirectionFile. Redirection File will be created
getLayerStack //Layer File was found in a higher Layer: "C:\\Users\\User\\AppData\\Local\\Temp\\ND7amS1xUnQJoFY7pLWXORoZBs40QeR8wMULXK8Kvi9d7vjz0N0X8Pl4qlQ0kMnp.json"

Seems there's some default debug logging enabled?

Also, I'm somewhat confused by those JSONs having the content:

{
    "data": null,
    "subLayers": []
}

Any examples on what to do with those files? What do the subLayers point to?

[BigRoy]

• In Houdini when pointing the Configure Stage Asset Resolver Context Path it crashes if you point it directly to a non-existing or existing .json file, and also crashes if you point it to a .usda file that doesnt have the required custom layer data.

• Triggering reload on the redirection file RedirectionFile.reload instance crashes houdini. For example this in a Python LOP:

from usdAssetResolver import AyonUsdResolver

node = hou.pwd()
stage = node.editableStage()

ctx = stage.GetPathResolverContext().Get()[0]
rd = ctx.getRedirectionFile()
rd.reload()   # <- crashes

All those cases should not crash Houdini.

• Adding a redirection through the Python API and saving the JSON file removes the original entries:

from usdAssetResolver import AyonUsdResolver

node = hou.pwd()
stage = node.editableStage()

ctx = stage.GetPathResolverContext().Get()[0]
rdf_file = ctx.getRedirectionFile()
rdf_file.addRedirection("missingRef.usda", "./sphere3.usda")
rdf_file.save()

• We're missing Python API functions from the Redirection File instance:
  • RedirectionFile.getRedirections
  • RedirectionFile.removeLayer
  • RedirectionFile.removeRedirection
  • RedirectionFile.clearLayers (remove all)
  • RedirectionFile.clearRedirections (remove all)

[Lypsolon]

Some context for this PR.

this pr implements a redirection system to allow informing an usd asset resolver about paths that should be at a different place at runtime.

its not the same as the pinning suport because pinning is suposed to be static by desing.

currently the system dose create files on disk as every rdf expects that it will be on disk.
this is a drawback from when it was first implemented and could be changed for better performance and less disk io.

the lakeFs branch for updates is here: lakefs://ayon-usd/rdf/

[BigRoy]

The latest version fixes the saving and reloading of the RDF file - so that does not crash anymore.

However, it still crashes if the configured asset file is invalid or does not exist. We should instead capture that with a warning and instead just continue as if there were no redirection file path set? We should at least just warn on invalid layers or files instead of crashing. The same goes for subLayers in the json.

We should at the very least avoid the resolver taking down e.g. Houdini with a hard crash.

Also, it seems if you query rdf_file.getDirections() it still lists entries that were removed in the current session, saved and reloaded. So after rdf_file.clearRedirections or rdf_file.removeRedirection it will still list those entries, even though it won't save them to a file on .save()