Current Projects

Robotics

We are working with a variety of robotic systems, focusing on mobility and the coupling of perception and control. We work with wheeled vehicles, flying machines, and legged robots. One of our emphases is on agile autonomy. We aim to enable deployment of autonomous mobile robots in previously unseen and unmapped environments. The robots should traverse these environments with no prior exposure to them, with purely onboard sensing and computation, and with great speed and agility. Much of our work leverages learning-based techniques, and develops unified treatments of perception and action.

Machine Learning

We are working on core models and algorithms in machine learning. One long-term interest is in deep network architectures. We are interested both in new layers and operators, and in new macroscopic structures and connectivity patterns. Another interest is the development of deep learning techniques for discrete structures, such as graphs and sets, and combinatorial optimization problems. Another line of work concerns settings such as continual learning, which depart from conventional offline batch supervised learning.

Computational Science

We are working on applications of machine learning to computational science. We are particularly interested in the use of deep networks and differentiable programming for simulation of physical systems. One line of work uses deep networks to predict the evolution of physical systems at high fidelity. Another makes physics simulators differentiable, so that deep networks that act on or reason about physical systems can be trained end-to-end by backpropagating through physics.

Sensorimotor Control and Simulation

We are working on sensorimotor control: learning to act based on raw sensory input. Our models and algorithms aim to support flexible operation in complex and dynamic three-dimensional environments. We are inspired by applications such as autonomous driving and household robotics, and by scientific curiosity. Much of our work leverages immersive simulation, and we have developed simulation platforms to support this field.

3D Representations

The world is three-dimensional. We are broadly interested in processing and synthesis of 3D data. One line of work concerns 3D reconstruction, with the aim of producing photorealistic models of objects and environments. Such photorealistic models are used in virtual reality, special effects production, and computer games, and can also be used in training visual perception and sensorimotor control systems. Another line of work concerns 3D data processing. We are interested both in classic optimization techniques and in novel deep network architectures that operate on three-dimensional representations.

Visual Perception

We are working on models and algorithms for visual perception. Our work has produced widely used network architectures and random field models for images. We have also created large-scale datasets and benchmarks for training and evaluating broad-competence visual perception systems. One interest is in robust visual perception: models that work reliably in the real world, across many environmental conditions, and can be rapidly deployed in new environments. Another line of inquiry is whether alternatives to convolutional networks can be even more effective than these standard models for images. Another long-term interest is video and how it can be used to enhance visual perception.

Image Processing and Image Synthesis

We are working on image processing and image synthesis. We have shown that deep networks can be used to quickly and accurately perform advanced image processing. We have developed deep network architectures that can synthesize photographic images, repeatedly advancing the realism of direct image synthesis. We have significantly advanced low-light imaging by applying deep networks directly to raw sensor data, in effect replacing much of the classic image processing pipeline; this technology has been widely adopted throughout the industry and is now in many smartphones. We are also introducing techniques for view synthesis with deep networks, allowing photorealistic exploration of complex large-scale scenes.


Past Projects

Low-Level Vision

We have worked on basic algorithmic building blocks that support visual data processing. Our algorithms have advanced the state of the art in optical flow estimation, visual odometry, bottom-up segmentation, intrinsic image decomposition, and other low-level vision problems.

Data-Driven 3D Modeling and Procedural Modeling

Making three-dimensional content creation easier is one of the main challenges in computer graphics. We have developed 3D modeling tools that maintain internal representations of the space of shapes that can be produced. This is enabled by representations for complex object categories, such as furniture, vehicles, and buildings. Can the space of all chairs, airplanes, or single-family homes be characterized computationally? Can generative models be designed that will synthesize custom chairs, airplanes, and houses according to specifications? Our work answers these questions affirmatively. Our representations and algorithms enable optimizing a variety of criteria over complex object classes and synthesizing novel instances from these classes.

Simulation of Human Motion

We have shown that high-fidelity human motion can be synthesized from first principles, given only a model of the human body and a compact set of objectives. For example, our techniques produce high-fidelity walking motion given a human model and the goal of advancing the center of mass forward at a specified velocity. The same techniques produce realistic running when the desired velocity is increased, without any other modifications to the objective or the model. We have shown that such de novo optimization can predict human motion both qualitatively and quantitatively.