Blend shapes (ARKit + Oculus visemes) and dynamic bones

[met4citizen]

Hi. I am working on the TalkingHead project, which provides a browser-based JavaScript class for real-time lip sync using 3D full-body avatars. Demo videos and use cases are available in the project README.

I'm new to MakeHuman/MPFB, but I created a character using MPFB and exported it from Blender for use with TalkingHead. After some post-processing and minor adjustments (renaming some objects/bones, adding LeftEye/RightEye bones, cleaning up materials for GLB/Three.js/WebGL, etc.), the character worked pretty well in the TalkingHead test web app.

The most time-consuming step was creating the facial blend shapes required for lip sync and facial expressions. Using the Faceit Blender add-on, this process took only about 15 minutes, but relying on a commercial add-on is not ideal, as it would require developers and/or end users to purchase additional software.

This leads me to wonder whether you might consider adding native support for ARKit and Oculus visemes blend shapes in some future releases?

This feature request is likely related to #169. - As I understand it, it is possible to create morph targets (aka blend shapes) in MPFB, but would these need to be created separately for each base mesh / facial topology and separately for each available asset (such as teeth, tongue, eyebrows, eyelashes)?

For what it is worth, creating blend shapes with Faceit does not require much artistic skill. The process mainly involves registering the relevant objects and placing landmarks. Based on this information, the add-on creates a facial rig and automatically generates the blend shapes using it. If reusable targets would indeed be the right way to implement this in MPFB, Faceit add-on might be useful in assisting with the initial creation.

In addition to blend shapes, I was also wondering whether assets can include extra bones and weights? - The reason for this question is that, in addition to text-driven and audio-driven lip sync, the TalkingHead class includes a lightweight physics engine that enables real-time physics simulation for additional "dynamic" bones, such as hair bones.

[joepal1976]

This leads me to wonder whether you might consider adding native support for ARKit and Oculus visemes blend shapes in some future releases?

As I have no experience with these, it seems unlikely I'll add it myself. But if someone wants to contribute such functionality, I'm all for it.

This feature request is likely related to #169. - As I understand it, it is possible to create morph targets (aka blend shapes) in MPFB, but would these need to be created separately for each base mesh / facial topology and separately for each available asset (such as teeth, tongue, eyebrows, eyelashes)?

No, targets only need to be modeled for the basemesh. In blender, all child meshes will adapt according to how they are matched to the base mesh at production. This is what happens when you either click the "refit assets" button or check the "auto refit assets" checkbox on the model panel.

If you export to a different environment, you'd have to replicate this functionality in order to recalculate the shape of child meshes when the basemesh changes. All logic for this is available in this file: https://github.com/makehumancommunity/mpfb2/blob/master/src/mpfb/services/targetservice.py

For basic docs on targets, see:

• Quick intro: https://static.makehumancommunity.org/mpfb/docs/assets/concept_targets.html
• On how to create: https://static.makehumancommunity.org/mpfb/docs/assets/creating_a_target.html

All in all, the main reason visemes and lip syncing hasn't been added is more a question of priorities than there being anything fundamental standing in the way. A good start would be to have free viseme targets available under a compatible license, but I haven't really looked into adding user interfaces and logic for loading and animating such visemes.

Particularly visemes which would require a recalculation of child meshes on each frame might require some thinking in order to make usable.

In addition to blend shapes, I was also wondering whether assets can include extra bones and weights? - The reason for this question is that, in addition to text-driven and audio-driven lip sync, the TalkingHead class includes a lightweight physics engine that enables real-time physics simulation for additional "dynamic" bones, such as hair bones.

Child meshes already support separate rigs and separate weights. The UX for creating these isn't necessarily for the faint of heart, but in principle the functionality is available. There is a tutorial for rigging the ponytail hair in the system assets here: https://static.makehumancommunity.org/mpfb/docs/rigging_mesh_assets.html

To my knowledge, this works, but it was a while since I tried it.

[met4citizen]

Thank you. I appreciate your detailed answers. They were very helpful. I went through the links you provided and spent some time experimenting with creating new targets in MPFB. I have one follow-up question:

As far as I know, shape keys on one mesh cannot directly drive shape keys on another mesh. In Blender, this can be done indirectly using drivers, and in the TalkingHead class we do it in code. However, in both cases, the same shape key must exist on both the base mesh and all relevant sub-meshes that need to deform.

Let’s take the Oculus blend shape "viseme_aa" as an example. In MPFB, I can create a new "viseme_aa" target, implement it on the base mesh by opening the lips appropriately, and save the target. I actually tried this, and everything works nicely. However, moving the lips alone is not sufficient. When pronouncing the phoneme associated with "viseme_aa", the teeth also open. To achieve this, I need to implement the same shape key, "viseme_aa", on the teeth mesh as well. - As I understand it, this is not something that can be saved as an MPFB target?

Maybe I am missing something, but I believe this issue also applies to eyes, eyebrows, eyelashes, teeth, and the tongue if I were to implement ARKit blend shapes such as "eyeLookDownLeft", "browDownLeft", "eyeBlinkLeft", "jawOpen", "tongueOut" etc. We need the same shape key to be implemented on all relevant meshes that need to deform.

You likely know how this works better than I do (I'm still learning Blender), but as a reference, you can download our example GLB avatar (./avatars/brunette.glb) and import it into Blender to see how the ARKit and Oculus blend shapes have been implemented there. In the TalkingHead project, developers have used several avatars / avatar creators (Ready Player Me, PlayerZero, AvaTurn, Avatar SDK, CC4, RocketBox, etc.), and this is how ARKit and Oculus visemes have been implemented across all of them.

[joepal1976]

What you are missing here are the "helpers": https://static.makehumancommunity.org/mpfb/docs/assets/concept_basemesh_and_helpers.html

The basemesh does not only consist of the body per se. There are also hidden mesh layers for fitting clothes, hair and bodyparts to.

To see these, disable the mask modifier on the base mesh.

When modeling a target which should have an influence on, for example, teeth, then you would make the target affect the vertices in the groups helper-upper-teeth and helper-lower-teeth.

[met4citizen]

Thank you. Got it.

These “helpers” seem to be a very clever approach. On the other hand, they are quite different from what we are doing in TalkingHead. The hope is that it would be possible to break down these targets+helpers into separate blend keys, so that the final exported model would fit easily into our existing real-time lip-sync and facial animation pipeline.

At this point, I have many new questions, but rather than taking your time covering the basics, I will read through the full documentation first and try out some of the more advanced features myself. That will likely take some time, but if I still have follow-ups afterward, I will open a new issue.

[met4citizen]

I created ARKit and two sets of viseme targets using Faceit. You can download the zip file of targets here: ARKit-and-visemes.zip.

When using a Rigify rig, the generation process was extremely straightforward. No need to set any landmarks. Below is a complete list of the steps:

MPFB2:

• New Human | From Scratch | Create Human
• Rigging | Add Rig | Add Rigify Rig | Generate

Faceit 2.3:

• Select the base mesh
• Setup | Register Face Objects | Select "Use Existing Face Rig"
• Expressions | Create | Load Faceit Expressions | Select "ARKit" | OK
• Expressions | Load Faceit Expressions | Select "Phonemes" and "Append" | OK
• Bake tab | Bake Shape Keys | Select "Action" and "None" | OK

As a result, I obtained 52 ARKit blend shapes (standard naming) and 21 Microsoft-compatible visemes (named *_01 - *_21) on the base mesh. I then added an empty blend shape for silence (sil_00) and saved all targets individually using Developer | Save Target. Finally, I duplicated a subset of the 22 Microsoft visemes into 15 Oculus/Meta visemes (name viseme_*). In total, this resulted in 52 + 22 + 15 = 89 targets.

Testing example:

• Create a new human with a Mixamo rig and some assets
• Developer | Load Targets | Select all, set weight to 0 | Load Targets
• Select the base mesh
• Change the value of "jawOpen" from 0 to 0.5
• Model | Refit Assets to Basemesh

Visually, everything appears to work as expected. The results are actually quite good, considering that I used the default Faceit settings without any fine-tuning or corrective work.

I would like to test these targets using my TalkingHead test app. However, as mentioned previously, I currently do not know how to bake them into new shape keys on all other relevant meshes. As you know, GLB only supports explicit morph targets per mesh and does not export deformation modifiers that reference other objects. As a result, these targets have no practical use for me at the moment.

As a more general note, typically these "targets" are used in MPFB as a way to design the character, and once the character is ready, they can in most cases be baked in terms of keeping the deformation but removing the blend key. The purpose of ARKit shapes and visemes is different. We usually need them only when the character is already finalized, and we don't want to bake and remove them at the end. Furthermore, while refitting is a good approach during character design, in use cases like ours where morph targets are updated in real time at 60 FPS, refitting is not a viable option. For these reasons, I believe these types of targets should be handled somewhat differently from standard design-time targets.

[joepal1976]

Disregarding the UX aspects of things, interpolating a shape key to a child mesh should not be very complicated.

You have two important pieces of info: The target file and the mhclo file.

The target specifies how much each vertex should shift in XYZ direction when applied 100%

The mhclo file specifies which three basemesh vertices should be considered and to what extent ("weight") when fitting the child mesh.

Thus combining the three basemesh vertex shifts multiplied by their individual weights would result in exact info on how much the child mesh vertex should shift with the target. (this is just me theorizing atm, there might be a flaw in this logic which might require an additional calculation)

For the baking shape keys on base mesh part but keeping visemes, this should not be very complicated either. It's mainly a question of first baking all modeling shape keys and then adding the viseme shapekeys afterwards.

So the basic operation of a theoretical new operator would be:

• Bake shape keys on basemesh
• Add viseme shape keys on basemesh
• Iterate over child meshes and interpolate viseme shape keys to these

Obviously there's a bit to discuss on how to make this accessible for users etc, but the core logic need not be that intimidating.