Procedural Publishing: AYON Publish ROP

[Lypsolon]

Changelog Description

Implements a special AYON Publish ROP node that publish a product from any of the files of its input ROPs. This way, any ROP can be easily published into a product type of the user's choice and it can act as a generic building block to allow publishing from within custom studio HDAs.

Additional review information

This PR implements a General Publish node into the /out context. The idea is that you will be capable to publish everything you want.

The approach

A new AYON Publish ROP will publish any of its direct inputs as representations to a new product defined by the AYON Publish ROP's attributes.

With systems like deadline's own submitter from Houdini you can then publish complex graphs to Deadline without requiring AYON logic to do the farm submission itself.

Implementation.

• A DriverRop node is used with pre render scripts
• We avoid problems with the SOHO system by having an single SOHO py file with a simple pass
• We use the Pre-Render script to call a function on the HDA python module. This function only exists to call the ayon_publish() function in the ayon_houdini module to keep most code in the AYON codebase instead of the HDA.
• The ayon_publish function calls publish with a selection of pyblish plugins
  • TODO: We may want to avoid doing this too much.
• When publishing from the GUI we will call the .render() function on the given ayon_publish node. In case we publish from a node we will set a variable to bypass the .render() call because the nodes will call it themselves.

Known issues

• Connecting multiple nodes that publish the same file extension this will be registered as separate representations and thereby error because the representations would overlap/overwrite each other.
• When you publish via AYON Publish nodes this will not increment your scene file because if we increment the $HIP variable gets out of sync because upstream nodes will export with the different file version as downstream nodes.
• When using the OpenGl node you will need to set an output image (ip or md does not work because it writes no files)
• Deadline submissions allows you to run every node on its set node ranged instead of respecting the deadline node range. (this is different than Houdini's behavior)
  • • in Houdini if the last node is on single frame all upstream nodes will be set to export only a single frame
  • our system will find out what the maximum is your node can export then it will check whats on disk and publish this
    • This avoids publishing "extra" files that should not be there
    • In case your node right now only publishes 1 frame but did publish 10 frames a min ago and the range is still set to 10 frames this system will end up publishing the extra 9 frames even if they don't belong there.

Testing notes:

Local publish

1. Launch Houdini via AYON.
2. Create the AYON Publish ROP
3. Connect some unique input file types (different extensions)
4. Each input would become a representation of the published product
5. Publish should work locally

Farm publish

It should work also completely on the farm when customizing some logic in Deadline's submitter, see the changes here.

Complex dependency graphs

A complex dependency graph in /out should also work fine. See the example files shared here.

---

TODO

• ensure that the hda has all the attributes
• Clean up the parameter layout for the ROP.
  • Scripts TAB should be at the bottom like the other ROPs
    • can they just be hidden ?
  • The SOHO parms should be invisible parms.
  • When rendering the ROP and an error occurred it would be nice to display the error on the ROP itself, like it turns red, etc. likely by raising an error explicitly in the code or by calling node.addError? (Do we need to have our own flag error event for this as we need to also checked on the error during cooking too.)
  • Make it very easy to select product types on the ROP or the publisher UI
  • Make sure publishing via publisher UI does not error on model validations if publish model product type (this may required a separate PR to differentiate the relevant validators for certain ROPs to target the ROPs instead once Show publish plug-in attributes based on families instead of only productType ayon-core#994 is merged.
  • Do not store AYON_productName on the instance, but compute it at runtime from creator + variant, etc.
  • Allow to get the JSON publish report from the 'headless' publish (unfortunately this requires a feature in ayon-core first: Create Context / Publish: Allow access to export the JSON report from CLI publish ayon-core#946

Use cases

- [ ] Showcase/demo how this could be used with e.g. a `FileCache` type node, including custom ones by a studio (we can test with the one from SideFX LABs)

[BigRoy]

Some mostly cosmetic notes - will test out functionality tomorrow.

[Lypsolon]

sooo here is a small scene to play around.

Important

its important to know that currently publishing only works via the render button on the rop. you can not trigger the setup from the Publisher Ui.

also all files need unique file extensions per ayon_publish rop node.

appendix

UT_PDG_Pub_PDG_GetAllData_v043.zip

[BigRoy]

@Lypsolon I checked out your PR.. decent start. I did some tweaking as I tested it: #127

The PR also fixes:

• find out how to correctly split .bgeo.sc extention from an houdini file name like UT _PDG_Pub_PDG_GetAllData_v030.geometry1.1001.bgeo.sc

• support more rops

We'll need to keep in mind some other goals to clean things up down the line:

• Clean up the parameter layout for the ROP.
  • Scripts TAB should be at the bottom like the other ROPs
  • The SOHO parms should be invisible parms.
  • When rendering the ROP and an error occurred it would be nice to display the error on the ROP itself, like it turns red, etc. likely by raising an error explicitly in the code or by calling node.addError?

These checkboxes have moved to the PR description instead.

[Lypsolon]

soo a bit later we got a little system that exports an advanced character via deadline.
i will append a scene that i used for experimentation.

its important to know that this system dose not support Ayon submit to farm right now as the submit to farm dose not generate the dependent jobs.

also the system will run Jobs for all the nodes in the Graph (as Deadline usually dose) this is nice because upstream nodes are executed and if we would do things like Run node A -> load the output -> export the loaded data this should work.
but i have the feeling that things are run multiple times (i have no prove for this but that's what i think)

UT_Roy_Test_Generic_v028.zip

[MustafaJafar]

Hello,
I've some cosmetic notes to be consistent with the current AYON HDAs if you don't mind.

1. This HDA file ayon-houdini\dell\ynput.dev.ayon_publish.1.6.hda, I think you can remove it from this PR. You can keep it on your side if you are using it, but can just exclude it when using git add

   git add .
   git restore --staged dell/ynput.dev.ayon_publish.1.6.hda
   

2. when saving/creating an HDA for ayon-houdini, we

• set the menu Entry to AYON
• set the Author to ayon
• set version to 1.0. currently, we don't change the version unless we have made a non backward compatible changes. (actual version control is done explicitly with git)
• don't use branches or adding them to the HDA name.
• add the Houdini context to the node. so, the name should be ayon_rop_publish
  

[Lypsolon]

1. This HDA file `ayon-houdini\dell\ynput.dev.ayon_publish.1.6.hda`, I think you can remove it from this PR. You can keep it on your side if you are using it, but can just exclude it  when using `git add`
   ```
   git add .
   git restore --staged dell/ynput.dev.ayon_publish.1.6.hda
   ```

forgot to remove it. did.

2. when saving/creating an HDA for ayon-houdini, we


* set the menu Entry to `AYON`

* set the Author to `ayon`

* set version to `1.0`. currently, we don't change the version unless we have made a non backward compatible changes. (actual version control is done explicitly with git)

* don't use branches or adding them to the HDA name.

* add the Houdini context to the node. so, the name should be `ayon_rop_publish`

done.

[moonyuet]

I have tested with pubish node <- really cool first of all. We hit some errors for getting product_type from the instance data.
We should set product_type as instance.data in the creator or use instance.data["productType"] inside the collect_output_node.py

Look forward the combination of two extensions for loaders btw.

[BigRoy]

Wasn't addError only allowed to be used from within a Python node or within the cook itself? See docs

This should only be called on the currently cooking node within a cook.

[moonyuet]

Nice catch. We need to write our own flag error instead of this then.