The Relightable Free-Viewpoint Video Project

Primary investigators

Christian Theobalt, Naveed Ahmed

---

High-quality Relightable Free-Viewpoint Video using a Skinned Template Model

Scope of the project:

By means of passive optical motion capture real people can be authentically animated and photo-realistically textured. To import real-world characters into virtual environments, however, also surface reflectance properties must be known. We describe a video-based modeling approach that captures human shape and motion as well as reflectance characteristics from a handful of synchronized video recordings. The presented method is able to recover spatially varying surface reflectance properties of clothes from multi-view video footage. The resulting model description enables us to realistically reproduce the appearance of animated virtual actors under different lighting conditions, as well as to interchange surface attributes among different people, e.g. for virtual dressing. Our contribution can be used to create 3D renditions of real-world people under arbitrary novel lighting conditions on standard graphics hardware.

Reconstruction improvement through cloth shift compensation	Overview of Spatio-temporal Dynamic Reflectance Sampling Procedure

In the context of the project, we developed a variety of spatio-temporal registration techniques, for instance to properly reconstruct the spatio-temporal appearance of apparel that is shifting while the person is moving. We also developed novel reflecance sharing concepts to robustly reconstruct dynamic surface reflectance even if there is a bias in the measured data.

Movie 1 - full overview (TVCG 2007)

Improvements through Spatio-temporal Reflectance Sharing (MIRAGE 2007)

Related Publications

• N Ahmed, C. Theobalt, M. Magnor, H.-P. Seidel, Spatio-Temporal Registration Techniques for Relightable 3D Video, Proc. of IEEE ICIP, 2007, to appear.


• N. Ahmed, C. Theobalt, H.-P. Seidel, Spatio-temporal Reflectance Sharing for Relighatble 3D Video, in Proc. of MIRAGE, Paris, 2007. [paper]


• C. Theobalt, N. Ahmed, H. Lensch, M. Magnor, H.-P. Seidel, Seeing People in Different Light: Joint Shape, Motion, and Reflectance Capture, in IEEE Transactions on Visualization and Computer Graphics, 13(4), p. 663-674 2007. [publisher's webpage][PDF]

C. Theobalt, N. Ahmed, H. Lensch, M. Magnor, H.-P. Seidel, Enhanced Dynamic Reflectometry for Relightable Free-Viewpoint Video, MPI-I-2006-4-006, 2006.

We have augmented our original model-based pipeline to capture and render free-viewpoint videos such that we do not only estimate dynamic texture information but also dynamic surface reflectance properties. This way, free-viewpoint videos can also be rendered under novel virtual lighting conditions.

Related Publications

• C. Theobalt, N. Ahmed, E. de Aguiar, G. Ziegler, H. Lensch, M. Magnor, H.-P. Seidel, Joint Motion and Reflectance Capture for Relightable 3D Video , Technical Sketch, ACM SIGGRAPH, Los Angeles, 2005. [PDF]
• C. Theobalt, N. Ahmed, E. de Aguiar, G. Ziegler, H. Lensch, M. Magnor, H.-P. Seidel, Joint Motion and Reflectance Capture for Creating Relightable 3D Videos, Technical Report MPI-I-2005-4-004, Max-Planck-Institut fuer Informatik, 2005, (PDF).

Contact