Neural Reconstruction and Rendering

Abstract

Digitizing the world around us is of increasing importance, with several applications in Extended Reality, movie and media production, telecommunications, medicine, video games, robotics, and many more. Digitisation involves three main stages: modeling, reconstruction and rendering. Modeling is the process of describing the semantics of real world objects and scenes. Reconstruction is the task of fitting the learned model to unseen data during test. The final step is rendering, where the reconstructed model is projected onto a 2D plane that represents the medium of observation e.g. screen, image, etc… Here, rendering that seamlessly blends with the environment is important for high-quality photorealism. In this talk, I will discuss my work on digitising our world through neural based approaches. I will cover all stages of the digitisation pipeline and will discuss how to process different types of input data modalities. I will cover several means of model building including supervised and self-supervised learning. I will also highlight how generative models can boost the learning capabilities and reduce reliance on paired training data. I will discuss recent advances in implicit scene representations and show the advantages they can bring over traditional meshes in a number of problems. In the end, I will highlight event cameras and the potential they can bring to 3D scene reconstruction and rendering. The ultimate goal of my work is to fully digitize our surroundings to allow all of us to connect and interact with our friends, family, and loved ones, over a distance.

For more info, please follow this link.