USD Resolver and pinning mode

[MustafaJafar]

Is there an existing issue for this?

• I have searched the existing issues and added correct labels.

Description

Overview

In short, IMO, the current state of pinning mode is more of a proof of concept.
Therefore, I'm making this issue to list out my findings, feel free to correct me if I'm wrong. we may create sub-issues of it.
Note, although pinning is used in the issue name, this issue is more about static cache remapping/override asset map file.

Current Behavior

Pinning File

• Pinning generation: the generated URI format is not compatible with AYON's URIs.
  Generated URI:

  ayon+entity:\\Experiments\\ideas\\test_resolver_pinning?product=worn_plaster_wall_diff_2k&version=v001&representation=jpg
  

  AYON's URI:

  ayon+entity://Experiments/ideas/test_resolver_pinning?product=worn_plaster_wall_diff_2k&version=v001&representation=jpg
  

• ayon_pinning_data_entry_scene item in pinning file is not a requirement for the pinning mode to function. it's even removed here . However, It can be useful to reflect which usd layer was used to generate the file. and it can be used for testing as mentioned here. so, you'd do something like:

  import json 
  from pxr import Usd
  
  # Open Stage
  parent_path = "E:/Ynput/tests/test_resolver_pinning_mode"
  pinning_file = f"{parent_path}/output_file.json"
  
  with open(pinning_file) as f:
      pinning_data = json.load(f)
  
      stage = Usd.Stage.Open(pinning_data["ayon_resolver_pinning_data"]["ayon_pinning_data_entry_scene"])
      print(stage.ExportToString())

• Multiple platforms: The pinning file generation doesn't substitute the roots with AYON template keys like {root[work]}
  and, the generated file is based on the machine you publish from. which won't work on machines with other operating systems.

Resolver

• Pinning mode is strict, it doesn't resolve any paths that are not in the pinning file
  therefore, opening Houdini workfiles in pinning mode casues seeing warnings like

  "Warning:     Could not load sublayer @c:/PROGRA~1/SIDEEF~1/HOUDIN~1.370/houdini/FreeCamera.usda@ of layer @anon:0000000F75C16200:tmp.usda@; skipping. (Recomposing stage on stage @anon:0000000F75C16200:tmp.usda@ <000000005A79F200>)"
  No camera defined in FreeCamera.usda files.
      C:/PROGRA~1/SIDEEF~1/HOUDIN~1.370/houdini/FreeCamera.usda
  

• When loading any USD products, Houdini's fallback values are not created or set resulting in textures don't show up even if the paths are resolved correctly.
  This could be the schemas are not being correctly applied when launching Houdini in Pinned mode OR could be a side effect of the resolver being strict.
  
  it's worth mentioning that shaders appear again when setting info:implementationSource attribute to id (mimicking the fallback values.)
  

• There is inconsistency in resolving: e.g. when having a relative path in your USD layer. the resolver converts it to absolute paths before resolving from the pinning file.

  • if the USD layer has ./worn_plaster_wall_2k/worn_plaster_wall_diff_2k.jpg and, the pinning file should have

    "h:/AYON/projects/Experiments/ideas/test_resolver_pinning/publish/usd/usdBox/v001/worn_plaster_wall_2k/worn_plaster_wall_diff_2k.jpg": "H:/AYON/projects/Experiments/ideas/test_resolver_pinning/publish/usd/usdBox/v001/worn_plaster_wall_2k/worn_plaster_wall_diff_2k.jpg",
    

  • however, resolving this ./worn_plaster_wall_2k/worn_plaster_wall_diff_2k.jpg directly using python won't work because this asset identifier doesn't exist in the pinning file.

    resolverContextCache::getAsset: (./worn_plaster_wall_2k/worn_plaster_wall_diff_2k.jpg) 
    >>> Resolved path: 

    as it expects the pinning file to look like this

    "./worn_plaster_wall_2k/worn_plaster_wall_diff_2k.jpg": "H:/AYON/projects/Experiments/ideas/test_resolver_pinning/publish/usd/usdBox/v001/worn_plaster_wall_2k/worn_plaster_wall_diff_2k.jpg",
    

Expected Behavior

• Speaking of having static cache mechanism with override asset map file. some of us expect/prefer these modes to be present in the resolver.
  • live mode: enable connection to AYON server + file path resolver.
  • light override: if an asset identifier exists in the mapping file, use that value. otherwise, resolve as in live mode.
  • offline: if an asset identifier exists in the mapping file, use that value. disable AYON server connection+ enable file path resolver.
  • offline strict (pinning): if an asset identifier exists in the mapping file, use that value. otherwise, error. disable AYON Server + disable file path resolver.
• The pinning/override file should be cross platform. it should consider win, linux and mac.
• The pinning/override file should follow some standard.

Testing Notes

Here are how I tested on my side:

1. in launcher use terminal app and select Houdin env
2. set some env variables and launch houdini.

   set ENABLE_STATIC_GLOBAL_CACHE=True
   set PINNING_FILE_PATH=H:/AYON/projects/Experiments/ideas/test_resolver_pinning/publish/usd/usdBox/v001/Box_pinning_data.json
   houdinifx
   

3. In houdini you can test it in two ways:
   a. using houini nodes, try loading your usd layer maybe using our ayon asset loader hda.
   b. using python where you test a uri from the pinned file.

from pxr import Ar

resolver = Ar.GetResolver()

unresolved_path = "ayon+entity://Experiments/ideas/test_resolver_pinning?product=worn_plaster_wall_diff_2k&version=v001&representation=jpg"
resolved_path = resolver.Resolve(unresolved_path)
print(f">>> Resolved path: {resolved_path}")

Here's an example pinning file:

{
  "ayon_resolver_pinning_data": {
    "H:\\AYON\\projects\\Experiments\\ideas\\test_resolver_pinning\\publish\\usd\\usdBox\\v001\\Expr_test_resolver_pinning_usdBox_v001.usd": "Absolute paths with back slash \\ and uppercase driver letter work.",
    "h:\\AYON\\projects\\Experiments\\ideas\\test_resolver_pinning\\publish\\usd\\usdBox\\v001\\Expr_test_resolver_pinning_usdBox_v001.usd": "Absolute paths with back slash \\ and lowercase driver letter work.",
    "H:/AYON/projects/Experiments/ideas/test_resolver_pinning/publish/usd/usdBox/v001/Expr_test_resolver_pinning_usdBox_v001.usd": "Absolute paths with forward slash / work and uppercase driver letter work.",
    "h:/AYON/projects/Experiments/ideas/test_resolver_pinning/publish/usd/usdBox/v001/Expr_test_resolver_pinning_usdBox_v001.usd": "Absolute paths with forward slash / work and lowercase driver letter work.",
    ".\\v001\\Expr_test_resolver_pinning_usdBox_v001.usd": "Relative paths with back slash \\ work.",
    "./v001/Expr_test_resolver_pinning_usdBox_v001.usd": "Relative paths with forward slash / work.",
    "ayon+entity://Experiments/ideas/test_resolver_pinning?product=usdBox&version=v001&representation=usd": "entity uris work.",

    "ayon_pinning_data_entry_scene": "H:\\AYON\\projects\\Experiments\\ideas\\test_resolver_pinning\\publish\\usd\\usdBox\\v001\\Expr_test_resolver_pinning_usdBox_v001.usd"
  }
}

you might have guessed it.. but I've added the example pinning file above to reflect that each one of these are different... it's really up to your workfile, you may use these paths interchangeably. however, the resolver will see them as different keys. so, here's case where it can be confusing.

Steps To Reproduce:

No response

Additional context:

Version

Version of the server, ayon-core and the addon that's affected

What platform were you running when you found the bug?

Windows
Linux - Rocky9
Linux - Ubuntu
MacOS

You can also provide any additional information relevant to the issue.

Relevant log output:

[MustafaJafar]

Tagging @philippe-ynput @BigRoy @antirotor for visibility.