Make Avatars more realistic

[Mortezanavidi]

Hi, first of all i want to thank you for open sourcing this great project, i have already read other issues regarding the obstacles in the way to make the models more realistic ( using the same mesh and mixamo compatible bone structure as readyplayerme and having the arkit and oculus.

But i have tried to my best to create a better model from scratch but i was unsuccessful, either the structure is mixamo compatible but not compatible with the library ( i belive it has something to do with the geometry of the model, the second it get's imported and the idle animations kick in, i can see the idle animations are not normal at all ).

And the lips would not move at all even a bit when the lip sync gets triggered, now i wanted to ask, is there any kind of example model which is outside of websites like readyplayerme or avaturn that can be reproducible? ( we could use their mesh and the modules and just change the bone structure and texture )

[met4citizen]

Hi. Unfortunately, I don't have any realistic model that could be used as a template. As of now, I have only used the class with Ready Player Me, Avaturn, and Mixamo characters.

The idle poses used in the TalkingHead class are based on Mixamo poses, so you could try using Mixamo for rigging, and then see how your custom model works with Mixamo poses/animations on their site. If that works, you can download and rename the bones (basically just by removing the "mixamorig:" prefix).

Note that the available Mixamo characters don't come with the needed blend shapes (ARKit/Oculus), and without them facial expressions and lip-sync won't work. Additionally, most of their characters were not designed to support moving eyes and lips. Therefore, I suggest using RPM models as a reference/template.

I'm no expert in 3D modeling, but if you think you have everything in place and it still doesn't work, maybe you can give a link to your GLB file so that I (or someone else reading this) can take a closer look. Without the model, it's hard to say what the problem might be.

[adamiprinciples]

So I'm working on getting this working with characters created with Realusions Character Creator 4. It's not been easy but I'm pretty close now I think. The tricky parts have been mapping the meta visemes to CC4 standard and adding support for jaw bone rotation to some of them. The result is pretty decent and should unlock the way for more realistic character options. (Character can look much better than in this screenshot with better lighting etc)

[met4citizen]

This looks amazing! I haven't tried Reallusion CC myself, but now I guess I have to take a closer look. Well done, Adam! I can't wait to see and hear more. I have received many requests for more realistic avatars, so I'm sure I'm not the only one interested.

[adamiprinciples]

What I am struggling with is removing all of your pose stuff without breaking everything. Ideally I just want to use the pose that comes with the model as my plan is to just loop an external idle animation over the top. I've renamed all the bones but obviously the ready player me skeleton is quite different so things look a little odd

[met4citizen]

Yes, these CC models/rigs and their workflows are very different from RPM+Mixamo, so it might not be worthwhile to try to force them into the same code. Nevertheless, understanding what works and what doesn't is valuable information in itself, so I hope you keep us informed of your progress or if you decide to set up your own repo.

Regarding poses, one approach could be to auto-rig your CC model in Mixamo, although I have no idea how well that would turn out. On the other hand, if you already have an idle animation for the CC rig, then you're probably better off sticking with it in your use case.

[namnm]

@adamiprinciples how are you doing so far with the realistic models? I would love to see your final result!

[AkshatRastogi-1nC0re]

@adamiprinciples I want to work on this with you. Is it possible we could connect regarding this ?

[RobLucchi]

Hello @adamiprinciples ,

I hope you're doing well, Really Impressive work btw.

I was wondering if you could share an update on your progress or any tips you might have for someone trying to achieve similar results. Your insights would be incredibly valuable, and I’m sure many others in the community would appreciate it too.

Thank you.

[met4citizen]

If you are still looking for more realistic avatars, you might also want to check out Microsoft Rocketbox. Their models are a bit old, but they are now available under the MIT License, which is nice. The rig is slightly different, but the models come with ARKit blendshapes and Oculus visemes. If you know the basics of 3D modeling, it’s not that much work to make them compatible with the TalkingHead class. I just tried converting one of their models, and here's the result as a proof-of-concept:

https://youtu.be/1otBhGEoGGg

Here are the main steps needed:

1. Download the model: Get the Microsoft Rocketbox "facial" version along with the Texture folder and its contents.
2. Remove the existing rig using Blender: Import the model into Blender, apply all transforms to all objects, apply the pose as the rest pose, remove the rig, and export to FBX.
3. Auto-rig in Mixamo: Upload the rig-free FBX to Mixamo, auto-rig it, and download as an FBX with a T-pose.
4. Add/rename/fix in Blender: Import the rigged FBX into Blender, apply all transforms to all objects, rename bones and shape keys, add LeftEye/RightEye bones, and set the following for all meshes: material > surface > metallic = 0. Fine-tune the model, then export it to GLB.

You can rename the bones and shape keys manually, but I wrote a short script for Blender (just within the limits of my Python skills):

import bpy, re

# Shape key map for Microsoft Rocketbox
shapekeyMap = [
[ "AA_VI_00_Sil", "viseme_sil" ],
[ "AA_VI_01_PP", "viseme_PP" ],
[ "AA_VI_02_FF", "viseme_FF" ],
[ "AA_VI_03_TH", "viseme_TH" ],
[ "AA_VI_04_DD", "viseme_DD" ],
[ "AA_VI_05_KK", "viseme_kk" ],
[ "AA_VI_06_CH", "viseme_CH" ],
[ "AA_VI_07_SS", "viseme_SS" ],
[ "AA_VI_08_nn", "viseme_nn" ],
[ "AA_VI_09_RR", "viseme_RR" ],
[ "AA_VI_10_aa", "viseme_aa" ],
[ "AA_VI_11_E", "viseme_E" ],
[ "AA_VI_12_I", "viseme_I" ],
[ "AA_VI_13_O", "viseme_O" ],
[ "AA_VI_14_U", "viseme_U" ],
[ "AK_01_BrowDownLeft", "browDownLeft" ],
[ "AK_02_BrowDownRight", "browDownRight" ],
[ "AK_03_BrowInnerUp", "browInnerUp" ],
[ "AK_04_BrowOuterUpLeft", "browOuterUpLeft" ],
[ "AK_05_BrowOuterUpRight", "browOuterUpRight" ],
[ "AK_06_CheekPuff", "cheekPuff" ],
[ "AK_07_CheekSquintLeft", "cheekSquintLeft" ],
[ "AK_08_CheekSquintRight", "cheekSquintRight" ],
[ "AK_09_EyeBlinkLeft", "eyeBlinkLeft" ],
[ "AK_10_EyeBlinkRight", "eyeBlinkRight" ],
[ "AK_11_EyeLookDownLeft", "eyeLookDownLeft" ],
[ "AK_12_EyeLookDownRight", "eyeLookDownRight" ],
[ "AK_13_EyeLookInLeft", "eyeLookInLeft" ],
[ "AK_14_EyeLookInRight", "eyeLookInRight" ],
[ "AK_15_EyeLookOutLeft", "eyeLookOutLeft" ],
[ "AK_16_EyeLookOutRight", "eyeLookOutRight" ],
[ "AK_17_EyeLookUpLeft", "eyeLookUpLeft" ],
[ "AK_18_EyeLookUpRight", "eyeLookUpRight" ],
[ "AK_19_EyeSquintLeft", "eyeSquintLeft" ],
[ "AK_20_EyeSquintRight", "eyeSquintRight" ],
[ "AK_21_EyeWideLeft", "eyeWideLeft" ],
[ "AK_22_EyeWideRight", "eyeWideRight" ],
[ "AK_23_JawForward", "jawForward" ],
[ "AK_24_JawLeft", "jawLeft" ],
[ "AK_25_JawOpen", "jawOpen" ],
[ "AK_26_JawRight", "jawRight" ],
[ "AK_27_MouthClose", "mouthClose" ],
[ "AK_28_MouthDimpleLeft", "mouthDimpleLeft" ],
[ "AK_29_MouthDimpleRight", "mouthDimpleRight" ],
[ "AK_30_MouthFrownLeft", "mouthFrownLeft" ],
[ "AK_31_MouthFrownRight", "mouthFrownRight" ],
[ "AK_32_MouthFunnel", "mouthFunnel" ],
[ "AK_33_MouthLeft", "mouthLeft" ],
[ "AK_34_MouthLowerDownLeft", "mouthLowerDownLeft" ],
[ "AK_35_MouthLowerDownRight", "mouthLowerDownRight" ],
[ "AK_36_MouthPressLeft", "mouthPressLeft" ],
[ "AK_37_MouthPressRight", "mouthPressRight" ],
[ "AK_38_MouthPucker", "mouthPucker" ],
[ "AK_39_MouthRight", "mouthRight" ],
[ "AK_40_MouthRollLower", "mouthRollLower" ],
[ "AK_41_MouthRollUpper", "mouthRollUpper" ],
[ "AK_42_MouthShrugLower", "mouthShrugLower" ],
[ "AK_43_MouthShrugUpper", "mouthShrugUpper" ],
[ "AK_44_MouthSmileLeft", "mouthSmileLeft" ],
[ "AK_45_MouthSmileRight", "mouthSmileRight" ],
[ "AK_46_MouthStretchLeft", "mouthStretchLeft" ],
[ "AK_47_MouthStretchRight", "mouthStretchRight" ],
[ "AK_48_MouthUpperUpLeft", "mouthUpperUpLeft" ],
[ "AK_49_MouthUpperUpRight", "mouthUpperUpRight" ],
[ "AK_50_NoseSneerLeft", "noseSneerLeft" ],
[ "AK_51_NoseSneerRight", "noseSneerRight" ],
[ "AK_52_TongueOut", "tongueOut" ],
[ "AU_26_JawDrop", "mouthOpen" ],
[ "HB_07_MouthSmile", "mouthSmile" ],
[ "AU_45_Blink", "eyesClosed" ],
[ "AU_63_EyesUp", "eyesLookUp" ],
[ "AU_64_EyesDown", "eyesLookDown" ]
]

# Recursive traverse
def traverse(x):
    yield x
    if hasattr(x, 'children'):
        for c in x.children:
            yield from traverse(c)
        
# Traverse objects
for r in bpy.context.scene.objects:
    for o in traverse(r):
        if hasattr(o, 'data'):
            if hasattr(o.data, 'bones'):
                
                # Rename mixamo bones
                for b in o.data.bones:
                    print(b.name);
                    b.name = re.sub(r'mixamorig:', '', b.name)
                    
            if hasattr(o.data, 'shape_keys'):
                if hasattr(o.data.shape_keys, 'key_blocks'):
                    
                    # Rename shape keys
                    keys = o.data.shape_keys.key_blocks
                    for m in shapekeyMap:
                        idx = keys.find(m[0])
                        if idx != -1:
                            print(idx)
                            keys[idx].name = m[1] 

In principle, you could use the same procedure for CC models. However, the Character Creator 4 software only works on Windows (I have a Mac), and at least the free model I downloaded didn’t include Oculus viseme support. In fact, Reallusion seems to have made every effort (both technically and legally) to lock you into their product ecosystem. If you plan on using their products or models, be sure to read their content end-user license agreement carefully.

[vesper8]

@adamiprinciples did you ever get your realistic avatar to work with Talking Head? I'd be very interested to try it out. Any chance you could share some GLBs if you managed to get it working? Thanks!

[3bdop]

Hello @met4citizen, I’m really interested in this approach to creating a more realistic avatar. I’m currently trying to map the shape keys of Reallusion CC3+ avatars with RPM using the Rokoko plugin, but I noticed that some of the shape keys are missing from the RPM avatar — for example, the teeth aren’t included. Will this affect the final result in the TalkingHead?

Also, is there another way to map those missing keys?

[met4citizen]

Hi @3bdop I haven't used CC or Rokoko myself, but as far as I know, the Rokoko plugin is used for retargeting animations only. It doesn't replace the skeleton, it just transfers animations. If your goal is to actually change the rig, you will need to use some rigging tool (or use the Mixamo online service as described above).

Based on the screenshot, you are mapping bones, not blend shapes. If the source animation includes movement of the teeth bone, those bone rotations won't be retargeted to the RPM avatar, since RPM avatars don't have teeth bones.

PS. I'm not sure how you are planning to use these realistic CC avatars, but note that the CC end-user license is much more restrictive than the RPM license. I can't give legal advice, but last I checked, the standard CC license doesn’t allow you to use a CC avatar in web apps like these where the model is downloaded to the browser.