3D Scene Understanding with a RGB-D Camera

Abstract: Three-dimensional scene understanding is important for computer systems that respond to and/or interact with the physical world, such as robotic manipulation and autonomous navigatio. For example, they may need to estimate the 3D geometry of the surrounding space (e.g., in order to navigate without collisions) and/or to recognize the semantic categories of nearby objects (e.g., in order to interact with them appropriately). In this talk, I will describe recent work on 3D scene understanding by the 3D Vision Group at Princeton University. I will focus on three projects that infer 3D structural and semantic models of scenes from partial observations with a RGB-D camera. The first learns to infer depth (D) from color (RGB) in regions where the depth sensor provides no return (e.g., because surfaces are shiny or far away). The second learns to predict the 3D structure and semantics within volumes of space occluded from view (e.g., behind a table). The third learns to infer the 3D structure and semantics of the entire surrounding environment (i.e., inferring an annotated 360 degree panorama from a single image). For each project, I will discuss the problem formulation, scene representation, network architecture, dataset curation, and potential applications.

This is joint work with Angel X. Chang, Angela Dai, Kyle Genova, Maciej Halber, Matthias Niessner, Shuran Song, Fisher Yu, Andy Zeng, and Yinda Zhang.

Bio: Thomas Funkhouser is the David M. Siegel Professor of Computer Science at Princeton University. He received a PhD in computer science from UC Berkeley in 1993 and was a member of the technical staff at Bell Labs until 1997 before joining the faculty at Princeton. For most of his career, he focused on research problems in computer graphics, including foundational work on 3D shape retrieval, analysis, and modeling. His most recent research has focused on 3D scene understanding in computer vision and robotics. He has published more than 100 research papers and received several awards, including a ACM SIGGRAPH Computer Graphics Achievement Award, ACM SIGGRAPH Academy membership, NSF Career Award, Sloan Foundation Fellowship, Emerson Electric, E. Lawrence Keyes Faculty Advancement Award, and University Council Excellence in Teaching Awards.