Question for Mesh used for DeformingThings4D

[tykim5931]

Hello,

I have a question for mesh from DeformingThings4D.
As I've look into the detailed code of pinnochio_runner.py, and found out it handles .anime file (which is from the DeformingThings4D dataset.

However, when I tried to load the .anime file and run the pinnochio, it fails to run the DemoUI program.
I think this is because of the flawness in mesh, but repairing mesh with trimesh and pymesh library never worked.

Have you ever been into this issue? And if you have, could you share your expirence to solve this problem?
Below is the messege right before making the skeleton.out from pinnochio fails.

Running Command: /home/lucy/ssd/Codes/transfer4D/Code/src/Pinocchio/DemoUI/DemoUI /home/lucy/ssd/Codes/transfer4D/dataset/bear3EP_attack1/results/debug/debug/mesh_obj_0/0_mesh.obj -skel /home/lucy/ssd/Codes/transfer4D/dataset/bear3EP_attack1/results/debug/debug/skeleton/0.skel
Reading /home/lucy/ssd/Codes/transfer4D/dataset/bear3EP_attack1/results/debug/debug/mesh_obj_0/0_mesh.obj
Mesh integrity error: edges[i].twin >= 0 && edges[i].twin < es in line 654
Somehow read /home/lucy/ssd/Codes/transfer4D/dataset/bear3EP_attack1/results/debug/debug/mesh_obj_0/0_mesh.obj: 11292 vertices, 67548 edges

I guess the reason is because processed mesh is not watertight? There is a hole in the bear's eye

Thank you.

[shubhMaheshwari]

Yes, I did face this issue. One of the drawbacks of using Pinnochio is it only works for watertight meshes since it uses an half edge represtation for medial axis calculations.

My solution is to take the largest component from the .anime file use meshlab to fill the wholes. You can run the meshlab script using the command:

meshlabserver -i input.obj -o output.obj -s closeholes.mlx

Additionally you can use boolean modifiers in blender to take the union of all meshes.

[tykim5931]

Hello,

I hope you're doing well. I've found a solution for recovering the mesh so it can be effectively used with Pinocchio. Specifically, I've added the following lines to pinnochio_runner.py right before saving the mesh:

mesh.remove_duplicate_faces()
mesh.remove_unreferenced_vertices()
mesh.remove_degenerate_faces()

if not mesh.is_winding_consistent:
    mesh.fix_normals()

if not mesh.is_watertight:
    mesh.fill_holes()

mesh.remove_unreferenced_vertices()

print("After processing:", mesh.vertices.shape, mesh.faces.shape)

if largest_component:
    components = mesh.split(only_watertight=False)
    if len(components) > 1:
        mesh = max(components, key=lambda comp: len(comp.faces))

save_obj(Path(filename), mesh.vertices, mesh.faces)

If it isn't too much trouble, I would greatly appreciate it if you could share your experiences regarding skeleton embedding inference with Pinocchio. We have encountered issues related to scale and orientation during rigging.

I'm considering the possibility that this might be related to the camera intrinsics and extrinsics used by PyTorch3D (which may lead to wrong scale inferenced in NRR.py), as we're using PyTorch3D's MeshRasterizer to render depth videos instead of Blender EEVE.

Any insights or suggestions you could offer would be greatly appreciated.

Thank you very much!