Load Alembic cache and manage as datablock

[BigRoy]

Changelog Description

Load an Alembic and manage the datablock instead of containerizing the root object.

Additional info

This simplifies the loading logic a lot since we do not need to take care of the hierarchy of the objects and keep track of which are children, etc.

Also, on load I enable that it ALWAYS add a cache reader using always_add_cache_reader=True. This may be less performant (so beware) but it massively simplifies code and improves reliability to what the user may expect when "updating" the cache. It means that even if object transforms of mesh vertex caches didn't animate at first, but do animate (or have moved) in the updated alembic they actually follow along. This also fixes that on update non-animated objects will get their transforms updated if they (even though static) have moved between the alembic versions.

This PR is solely to test the concept - and discuss good things about this; and of course I'm looking forward to all the issues one may have with this. So, feedback is welcome!

Thanks to Pete Addington from The Line for derailing me to start fiddling with this.

Testing notes:

1. Use Load Cache Data loader for Alembics.
2. Play with it, update it, manage it, remove it.

Resolves #60

[moonyuet]

I tested with some files sent by Libor
It works, only thing which is not working is the switch function but I dont think we need to have that for the load alembic cache.
I believe we can remove the load_cache and load_camera_abc and simply use this loader to load camera and cache?
https://github.com/user-attachments/assets/58ef022c-6f4b-499c-adf3-3734d6a0d021

[BigRoy]

I tested with some files sent by Libor It works, only thing which is not working is the switch function but I dont think we need to have that for the load alembic cache. I believe we can remove the load_cache and load_camera_abc and simply use this loader to load camera and cache? https://github.com/user-attachments/assets/58ef022c-6f4b-499c-adf3-3734d6a0d021

I'll need to test that switching. Did it actually error? or just fail to show any changes to the scene? I suspect the "old" objects stop moving? (because their object paths are not found in the new alembic, likely objects are named differently in the other file?) And the new geometry does not get loaded? If so, then yes - new geometry does not get 'created' currently on update or switch. It should be doable, I'm just not sure how trivial and sensible it would be. As in, I'm not sure if it'd be 100% reliable and if sometimes it'd then start creating objects that maybe the user deleted on purpose. :D

[moonyuet]

I tested with some files sent by Libor It works, only thing which is not working is the switch function but I dont think we need to have that for the load alembic cache. I believe we can remove the load_cache and load_camera_abc and simply use this loader to load camera and cache? https://github.com/user-attachments/assets/58ef022c-6f4b-499c-adf3-3734d6a0d021

I'll need to test that switching. Did it actually error? or just fail to show any changes to the scene? I suspect the "old" objects stop moving? (because their object paths are not found in the new alembic, likely objects are named differently in the other file?) And the new geometry does not get loaded? If so, then yes - new geometry does not get 'created' currently on update or switch. It should be doable, I'm just not sure how trivial and sensible it would be. As in, I'm not sure if it'd be 100% reliable and if sometimes it'd then start creating objects that maybe the user deleted on purpose. :D

It doesn't error and it does switch but the object in the scene does not get correctly updated. And it is supposed to be working for that switch function since the update function of the filepath works as expected.

ayon-blender/client/ayon_blender/plugins/load/load_cache_data.py

Line 72 in a4a4193

cache_file.filepath = self.filepath_from_context(context)

[LiborBatek]

I have tested Load cache Data and it works...

Also my finding is that every node within Alembic gets transform cache constraint which then prevents from any tweaks of the transforms by user (solution is to set Influence of the transform to zero) - this doesnt mean its an issue, but something which is not normally happening aka no any transform constraints added when using Load cache (Abc)

Once overriden by turning the influence to zero, it holds the transforms user edit even when performing Manage>Set Version etc. which is good.

[BigRoy]

I have tested Load cache Data and it works...

Also my finding is that every node within Alembic gets transform cache constraint which then prevents from any tweaks of the transforms by user (solution is to set Influence of the transform to zero) - this doesnt mean its an issue, but something which is not normally happening aka no any transform constraints added when using Load cache (Abc)

Once overriden by turning the influence to zero, it holds the transforms user edit even when performing Manage>Set Version etc. which is good.

Thanks - yes.

I actually wanted to do potentially do a few more things:

• Add a parent group by default (however, since we're not tracking any objects/hierarchies... I'm not sure what should happen to the group on "remove" because we technically do not reliably know about that group itself). But the parent group would at least allow you to move the asset without load non-destructively
• Add a 'namespace' or other prefix to the loaded objects - so that the naming is clearer as to what was loaded. Which may be a customizable "preset" defined from settings.

[LiborBatek]

I got your points...sounds reasonable!

One note tho: we are facing opposite issue with super long names which got unneccessary loong once asset got loaded - something which should be taken care of - this been issue mostly when going for nested scenes as the names got clipped off.

So aiming for short and clear naming been the top priority speaking of blender integration... I will give some more info/ example of those issue connected asap.

Also unnecessary grouping (via locators/ empties) occurs a lot making lot of cluttering in the outliner...

[BigRoy]

So aiming for short and clear naming been the top priority speaking of blender integration... I will give some more info/ example of those issue connected asap.

Also unnecessary grouping (via locators/ empties) occurs a lot making lot of cluttering in the outliner...

Both valid points - then I'd be happy to keep it as barebones as is if there is no actual need in production for more uniqueness?

[LiborBatek]

Even tho its a bit off topic I will share it here...u can blame me later for sure :D

For e.g creating blendScene publish instance is resulting into this naming of blender Collection (acting as a product set in maya way)

its quite long as it uses folder productType and Variant ...maybe its just that blendScene type is tad long imho and could be something like blndScn or such.

This start to be an issue when loading such asset into animation scene and publishing layout from there using those loaded assets which can quickly result into massive long names when loaded into lighting scene similar to this:

So imho just optimizing it a bit could help a lot as this can be quite painful to navigate in.

just small update note: it should get better with recently merged PR dealing with stripping Namespace so nevermind it now

[moonyuet]

@BigRoy any update on this sponsored PR?

[LiborBatek]

@BigRoy anything to say regarding this PR...is it still valid or no? Will we migrate to this or not? Im really not sure whats going on with this one....and is a bit stale.

[BigRoy]

any update on this sponsored PR?

Extremely low prio for the client -hence... no time to spend on it really.

anything to say regarding this PR...is it still valid or no? Will we migrate to this or not? Im really not sure whats going on with this one....and is a bit stale.

I think it has more merit as a means of managing what's loaded than how we're currently dealing with loading Alembics - but seems there's too little interest to have things changed around by clients - as such, it remains at low prio and no time to spend on it.

Maybe @philippe-ynput can evaluate this workflow over the current Load Alembic logic in develop.

[moonyuet]

I think it has more merit as a means of managing what's loaded than how we're currently dealing with loading Alembics

I am only concerned with switching function from alembic camera and alembic assets, but it is just a minor thing. Otherwise, it looks good for me.

[BigRoy]

Non-critical, but:

Just want to remark that this way of loading may face the issue that "Edge/Vertex Creases" are NOT loaded by the MeshSequenceReader on the objects in the scene, see Blender issue: https://projects.blender.org/blender/blender/issues/141633

However, there is a pending PR in Blender that does work to fix that https://projects.blender.org/blender/blender/pulls/141646 so it may solve it in future versions.

[LiborBatek]

@BigRoy ...sorry but I have merged latest develop into this branch due to outdated 'AVALON' container used in this branch but probably broke it by that... as I was not able to load any rig and simply merged develop in....probably not best idea!

now giving me issue when trying set version via Manage>

  File "C:\Work\REPO\ayon-blender\client\ayon_blender\api\ops.py", line 129, in execute
    result = callback(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^

  File "C:\Work\REPO\ayon-blender\client\ayon_blender\plugins\load\load_cache_data.py", line 80, in exec_update
    cache_file[AVALON_PROPERTY]["representation"] = (
    ~~~~~~~~~~^^^^^^^^^^^^^^^^^

KeyError: 'bpy_struct[key]: key "avalon" not found'

Some good news tho:

I have tested anyway despite that error and it proved to be working for all occasions:

animation pointcache camera and I have also tested the scenarios with updating the Alembic with non movable objects suddenly animated in the newer version plus stationary object but changing its transforms among Alembic versions.

All seems good to me.

[LiborBatek]

@BigRoy one more question do we unify these entries in the Loader once merged? as Im not sure atm as there are multiple ABC loaders next to each:

[BigRoy]

one more question do we unify these entries in the Loader once merged? as Im not sure atm as there are multiple ABC loaders next to each:

Not entirely sure yet - there could be reasons for both - e.g. Cache Data I've never tried with loading a camera, does that even work? :) And... I'm not sure if we could e.g. auto-upgrade the old one? I guess we could at some point have enable settings for each loader and maybe also pick one of the two being disabled by default.

[LiborBatek]

Not entirely sure yet - there could be reasons for both - e.g. Cache Data I've never tried with loading a camera, does that even work? :) And... I'm not sure if we could e.g. auto-upgrade the old one? I guess we could at some point have enable settings for each loader and maybe also pick one of the two being disabled by default.

yes I have tested with camera product too and Load Cache Data works fine with it too.

Im for using the new load cache data as default (and naming it just as Load (Abc) ) and the other treat it as u have suggested above (I will leave it to you whats better way) ...simply resulting into single menu load action for Alembics

edited note: Maybe having that legacy loader as an option on the Load Cache Data which will be named Load (Abc)

similarly to Link Blend option box:

[LiborBatek]

@BigRoy @moonyuet is there anything to solve still? as it would be nice to get rid of this one finally...

[moonyuet]

@BigRoy @moonyuet is there anything to solve still? as it would be nice to get rid of this one finally...

I agree, i dont have any issue for this one. I am good to go, maybe ask @philippe-ynput

[LiborBatek]

Note: we should end up with single load action e.g. Load cache (Abc) to lessen possible confusion if using legacy and new type together

[LiborBatek]

@moonyuet so can we wrap it up or ? its hanging here for serious time and its working, right? I dont know whats the blocker here.

[LiborBatek]

@BigRoy how about that, sir?