Current Projects

Sensorimotor Control and Simulation

We are working on sensorimotor control: learning to act based on raw sensory input. Our models and algorithms aim to support flexible operation in complex and dynamic three-dimensional environments. We are inspired by applications such as autonomous driving and household robotics, and by scientific curiosity. Much of our work leverages immersive simulations, and we have developed simulation platforms to support this field.

3D Reconstruction and Geometry Processing

The world is three-dimensional. We are broadly interested in processing and synthesis of 3D data. One line of work concerns 3D reconstruction, with the aim of producing photorealistic models of objects and environments. Such photorealistic models are used in virtual reality, special effects production, and computer games, and can also be used in training visual perception and sensorimotor control systems. Another line of work concerns 3D data processing. We are interested both in classic optimization techniques and in novel deep network architectures that operate on three-dimensional representations.

Image Processing and Image Synthesis

We are working on image processing and image synthesis. We are particularly interested in novel deep network architectures. We have shown that deep networks can be used to quickly and accurately perform advanced image processing. We have also developed deep network architectures that can synthesize photographic images.

Scene Understanding

We are working on models and algorithms for visual perception. Our work has produced widely used convolutional network architectures and the dense random field model. We have also created large-scale datasets and benchmarks for training and evaluating broad-competence visual perception systems.

Low-Level Vision

We are working on basic algorithmic building blocks that support visual data processing. Our algorithms have advanced the state of the art in optical flow estimation, visual odometry, bottom-up segmentation, intrinsic image decomposition, and other low-level vision problems.

Past Projects

Data-Driven 3D Modeling and Procedural Modeling

Making three-dimensional content creation easier is one of the main challenges in computer graphics. We have developed 3D modeling tools that maintain internal representations of the space of shapes that can be produced. This is enabled by representations for complex object categories, such as furniture, vehicles, and buildings. Can the space of all chairs, airplanes, or single-family homes be characterized computationally? Can generative models be designed that will synthesize custom chairs, airplanes, and houses according to specifications? Our work answers these questions affirmatively. Our representations and algorithms enable optimizing a variety of criteria over complex object classes and synthesizing novel instances from these classes.

Simulation of Human Motion

We have shown that high-fidelity human motion can be synthesized from first principles, given only a model of the human body and a compact set of objectives. For example, our techniques produce high-fidelity walking motion given a human model and the goal of advancing the center of mass forward at a specified velocity. The same techniques produce realistic running when the desired velocity is increased, without any other modifications to the objective or the model. We have shown that such de novo optimization can predict human motion both qualitatively and quantitatively.