Real-Time Dynamic Rendering in VR

A comprehensive pipeline for real-time rendering of dynamic scenes in VR.

This repo contains mesh generation for dynamic objects, mesh simplification and unity rendering example scenes. The rendering targets Meta Oculus Quest 2, Unity Version: 2022.3.10f1.

1. DynamicMeshGeneration: This is a modified version of ResFields to enable the extraction of meshes with vertex colors of a dynamic scene from 2D input videos.

2. MeshProcessing: Python script for Blender. The extracted raw mesh sequence with vertex color will be processed by Blender and transformed to simplified meshes and corresponding texture maps (diffuse map and normal map).

3. DynamicRendering: Unity example scene and scripts.

   • The simplified mesh sequence, static mesh and background are put into one scene. There are also custom shaders and materials.
   • Some C# scripts are used to perform scene complexity statistics, load mesh sequence, assign custom material to object sequence, and animate mesh sequence with user tunable speed.

4. DynamicRenderingGSScene: We additionally provide unity rendering examples of Gaussian Splatting scene.

Mesh Generation

To generate meshes of static objects or scenes, we recommend using following Neural Reconstruction methods:

• NeuS
• Neuralangelo
• sdfstudio

For dynamic objects, we provide a modified version of ResFields under DynamicMeshGeneration. These modifications are designed to facilitate the extraction of meshes that carry vertex colors.

Mesh Processing

scripts with `import bpy` are run inside blender
other scripts: python3.8 and pymeshlab

1. surface_reconstruction/: scripts for surface resonstruction from point cloud

2. format_conversion/: scripts for format conversion, ply -> obj -> fbx.

3. background_edit/: script to trunk mesh outside the given region. This is used to remove the reconstruction artifacts in the background.

4. mesh_compression/

   • 1_mesh_downsample_blender.py: step 1, apply series of decimate modifiers to the original mesh and do uv unwrapping.
   • 2_texture_baking_blender.py: step 2, perform texture baking. First create a material that use vertex color as diffuse color for the original mesh. Then, perform texture baking for diffuse map and normal map.

5. mesh_psnr/:

   • Render the mesh using vertex color (1_render_vertex_color.py) and texture maps (2_render_textured_mesh.py) in Blender. Camera positions and lighting configs can be adjusted in the scripts.
   • 3_calculate_psnr_ssim.py: Calculate the PSNR and SSIM between the images(can choose to use full image or just part of the image, e.g. only the head).

Dynamic Rendering

The unity project to render the simplified the mesh, our target platform is Meta Oculus Quest 2. You may need to reimport the need package in your unity project(Here)

1. Our extracted/simplified meshes are under Assets/Meshes, demo scene: Assets/Scenes/DynamicRenderingDemo.unity

2. Our C# scripts and custom shaders are under Assets/DynamicRendering

   • Assets/DynamicRendering/Scripts: script to animate mesh sequence, basic interaction to addjust the animation speed and basic UI display for mesh speed and FPS.
   • Assets/DynamicRendering/Editor: Unity editor script. For scene complexity statistics, automatically import mesh sequence and apply custom material to all meshes in the sequence.
   • Assets/DynamicRendering/Shaders and Assets/DynamicRendering/Materials: Custom materials for vertex color only and shaders defined by shader graph.

Dynamic Rendering Gaussian Splatting Scene

To render Gaussian Splatting scenes, we refer users to DynamicRenderingGSScene/Readme.md.

Contributors

• Jiayong Li (jiayli@student.ethz.ch)
• Kaixi Bao (kaibao@student.ethz.ch)
• Yuhang Qiu (yuhqiu@student.ethz.ch)
• Zehang Qiu (zehqiu@student.ethz.ch)

Supervised by Marko Mihajlovic (marko.mihajlovic@inf.ethz.ch) and Sergey Prokudin (sergey.prokudin@inf.ethz.ch).