We are working on sensorimotor control in immersive environments.
We are pursuing two related lines of work. The first concerns 3D reconstruction: the creation of high-fidelity 3D models of objects and environments from visual data, such as images, video, and RGB-D video. 3D reconstruction provides data for virtual reality, special effects production, computer games, and other applications that utilize three-dimensional content. 3D models can also be used to enhance the performance of computer vision systems. The second line of work concerns algorithms and systems that process 3D data, including data produced by 3D reconstruction pipelines.
We are working on algorithms for understanding real-world scenes from visual input. Our work has yielded the widely used dense random field model and, more recently, new convolutional network architectures. We have also developed approaches to estimating the spatial layout of scenes.
Making three-dimensional content creation easier is one of the main challenges in computer graphics. We have developed 3D modeling tools that maintain internal representations of the space of shapes that can be produced. This is enabled by representations for complex object categories, such as furniture, vehicles, and buildings. Can the space of all chairs, airplanes, or single-family homes be characterized computationally? Can generative models be designed that will synthesize custom chairs, airplanes, and houses according to specifications? Our work answers these questions affirmatively. Our representations and algorithms enable optimizing a variety of criteria over complex object classes and synthesizing novel instances from these classes.
We have shown that high-fidelity human motion can be synthesized from first principles, given only a model of the human body and a compact set of objectives. For example, our techniques produce high-fidelity walking motion given a human model and the goal of advancing the center of mass forward at a specified velocity. The same techniques produce realistic running when the desired velocity is increased, without any other modifications to the objective or the model. We have shown that such de novo optimization can predict human motion both qualitatively and quantitatively.