News

Reaching the Limit in Autonomous Racing: Optimal Control versus Reinforcement Learning was published in Science Robotics and featured on the journal’s cover. The paper presents a systematic study of control system design methodologies, focusing on reinforcement learning and optimal control in the context of autonomous drone racing. We show that neural networks trained with reinforcement learning (RL) outperform optimal control methods and trace the root cause to the flexibility afforded by RL in the formulation of the controller’s objective. In conjunction with the study, we push autonomous drones to unprecedented performance regimes, demonstrating superhuman control while reaching accelerations greater than 12g and velocities greater than 100 km/h.

Champion-level Drone Racing using Deep Reinforcement Learning was published in Nature and featured on the journal’s cover. The paper presents an autonomous system that outraced human world champions in first-person view (FPV) drone racing, a televised sport. This is the first time that an autonomous mobile robot achieved world-champion-level performance in a real-world competitive physical sport. Here is a video created by Nature to highlight the work and here is an accompanying article that provides a good overview.

The paper Optimizing Locomotion Controllers Using Biologically-Based Actuators and Objectives received the ACM SIGGRAPH Test-of-Time Award at SIGGRAPH 2023. The award “recognizes highly influential papers published in SIGGRAPH conferences that have made a significant impact over the past 10 years or more”. This is the first year of this annual award, for which all papers presented at SIGGRAPH conferences from 2011 to 2013 were considered. Four papers were selected for the award.

Learning robust perceptive locomotion for quadrupedal robots in the wild was published in Science Robotics. We have combined proprioception with visual perception to create a locomotion controller with unprecedented versatility and robustness. The controller went on an hour-long hike up and down a mountain in Switzerland, without a single fall, completing the trail in the same time as recommended in the hiking guide for humans. It also powered all the legged robots for the winning team in the DARPA Subterranean Challenge. The key contribution is combining visual perception with proprioception. Everything is trained purely in simulation and transferred zero-shot to reality.

Learning high-speed flight in the wild was published in Science Robotics. The paper presents an end-to-end approach to autonomous high-speed flight through complex natural and human-made environments, with purely onboard sensing and computation. The flight is controlled by a convolutional network that maps noisy sensory observations to collision-free trajectories in a receding-horizon fashion. It is trained exclusively in simulation via privileged learning.

Five papers were accepted to the International Conference on Computer Vision (ICCV). Two of these were selected for full oral presentation at the conference (3.4% acceptance rate): Point Transformer and Learning to Drive from a World on Rails.

Learning Quadrupedal Locomotion over Challenging Terrain was published in Science Robotics and featured on the journal’s cover. The paper presents a radically robust legged locomotion controller for rough terrain and demonstrates remarkable zero-shot generalization from simulation to natural environments. The controller retains its robustness under conditions that have never been encountered during training: deformable terrain such as mud and snow, dynamic footholds such as rubble, and overground impediments such as thick vegetation and gushing water.

The paper Multiscale Deep Equilibrium Models was accepted to Neural Information Processing Systems (NeurIPS) and selected for oral presentation at the conference (1.1% acceptance rate).

Three papers were accepted to the European Conference on Computer Vision (ECCV): Free View Synthesis, Tracking Objects as Points, and Dynamic Low-light Imaging with Quanta Image Sensors. Tracking Objects as Points was selected for a spotlight presentation at the conference (5.3% acceptance rate).

Deep Drone Acrobatics was nominated for the Best Paper Award at the Robotics: Science and Systems (RSS) 2020 conference. It was one of 3 papers nominated for the award, out of 321 submissions.

Five papers were accepted to the Conference on Computer Vision and Pattern Recognition (CVPR). Deep Global Registration and High-dimensional Convolutional Networks were selected for oral presentation at the conference (5.7% acceptance rate).

Habitat: A Platform for Embodied AI Research was nominated for the Best Paper Award at the International Conference on Computer Vision (ICCV). It was one of 11 papers nominated or selected for awards at the conference, out of 4303 submissions.

Two papers were accepted to Neural Information Processing Systems (NeurIPS): Deep Equilibrium Models and Differentiable Cloth Simulation for Inverse Problems. Deep Equilibrium Models were selected for a spotlight presentation at the conference (3% acceptance rate).

Five papers were accepted to the International Conference on Computer Vision (ICCV). Three of these were selected for full oral presentation at the conference (4.6% acceptance rate): Seeing Motion in the Dark, Consensus Maximization Tree Search Revisited, and Habitat: A Platform for Embodied AI Research

Does Computer Vision Matter for Action? was published in Science Robotics. The paper studies whether explicit intermediate representations of the kind studied in computer vision help action. We probe this question via controlled experiments in immersive simulation. Our main finding is that computer vision does matter. Models equipped with intermediate representations train faster, achieve higher task performance, and generalize better to previously unseen environments.

Five papers were accepted to the Conference on Computer Vision and Pattern Recognition (CVPR). Acoustic Non-Line-of-Sight Imaging was selected for oral presentation at the conference (5.6% acceptance rate).

Learning Agile and Dynamic Motor Skills for Legged Robots was published in Science Robotics. The paper introduces a method for training a neural network policy in simulation and transferring it to a state-of-the-art legged system. The approach is applied to the ANYmal robot, a sophisticated medium-dog–sized quadrupedal machine. Using policies trained in simulation, the robot achieves locomotion skills that go beyond what had been achieved with prior methods: ANYmal is capable of precisely and energy-efficiently following high-level body velocity commands, running faster than before, and recovering from falling even in complex configurations.

Our paper Deep Drone Racing: Learning Agile Flight in Dynamic Environments received the Best Systems Paper Award at the Conference on Robot Learning (CoRL). Three papers were selected for awards of any kind, out of 237 submissions to the conference.

Three papers were accepted to the Conference on Robot Learning (CoRL): Deep Drone Racing, Driving Policy Transfer, and Motion Perception in Reinforcement Learning with Dynamic Objects. Deep Drone Racing was selected for full oral presentation at the conference.

Four papers were accepted to the Conference on Computer Vision and Pattern Recognition (CVPR). Semi-parametric Image Synthesis was selected for full oral presentation at the conference (2.1% acceptance rate). Tangent Convolutions for Dense Prediction in 3D was selected for a spotlight oral.

We have released Open3D, a modern open-source library for 3D data processing. Open3D has been in development and internal use for three years, has supported multiple published research projects, and is actively deployed in the cloud. It is released open-source under the permissive MIT license and is now available at www.open3d.org.

Robust Continuous Clustering was published in the Proceedings of the National Academy of Sciences (PNAS). The paper presents a clustering algorithm that optimizes a smooth global objective using efficient numerical methods. This allows clustering to be integrated into end-to-end feature learning pipelines. Our algorithm effectively untangles mixed clusters, achieves high accuracy across domains, and scales to high dimensions and large datasets.

Five papers were accepted to the International Conference on Computer Vision (ICCV). Photographic Image Synthesis was selected for full oral presentation at the conference (2.1% acceptance rate). Playing for Benchmarks was selected for a spotlight oral.

The paper Learning to Act by Predicting the Future was accepted to the International Conference on Learning Representations (ICLR) and selected for full oral presentation at the conference (3% acceptance rate).

Our entry won the Visual Doom AI Competition. The competition evaluates sensorimotor control models that act in three-dimensional environments based on raw sensory input. Our entry, IntelAct, placed first in the “Full deathmatch” track, which takes place in previously unseen environments. Our model was trained end-to-end: from pixels to actions. It learns to act from raw experience, without human gameplay or other “expert” supervision. The work is based on an approach to sensorimotor control that will be presented in a forthcoming research paper. A video preview is available.

Two papers were accepted to the European Conference on Computer Vision (ECCV): Playing for Data and Fast Global Registration. The latter was selected for full oral presentation at the conference (1.8% acceptance rate).

Three papers were accepted to the Conference on Computer Vision and Pattern Recognition (CVPR). Two of these were selected for full oral presentation at the conference (3.9% acceptance rate).

Adobe released a product based on our data-driven 3D modeling research at Stanford. The original research was described in papers published at SIGGRAPH Asia 2010 and SIGGRAPH 2011, and was the subject of Siddhartha Chaudhuri’s PhD thesis. Our technology was licensed in 2012 by Mixamo, a start-up company led by Stefano Corazza. Mixamo turned the technology into a product called Fuse. Fuse became a widely used 3D character modeling tool and led to Adobe’s acquisition of Mixamo earlier this year. Fuse has now been rebranded and launched as part of the Adobe Creative Cloud.

The paper Robust Nonrigid Registration by Convex Optimization was accepted to the International Conference on Computer Vision (ICCV) and selected for full oral presentation at the conference (3.3% acceptance rate).

Joined Intel Labs to build a new lab devoted to basic research.

The paper Geodesic Object Proposals was accepted to the European Conference on Computer Vision (ECCV) and selected for full oral presentation at the conference (2.8% acceptance rate).

I’m serving as a Program Chair of the Symposium on Computer Animation. The submission deadline is April 15. The call for papers is here.

Two papers were accepted to the International Conference on Computer Vision (ICCV): A Simple Model for Intrinsic Image Decomposition with Depth Cues and Elastic Fragments for Dense Scene Reconstruction. The latter was selected for full oral presentation at the conference (2.5% acceptance rate).

Mixamo released a character modeling tool based on our assembly-based modeling work. The modeling tool is called Fuse and is a polished version of the system developed by Siddhartha Chaudhuri for his PhD thesis.

The paper Efficient Nonlocal Regularization for Optical Flow was accepted to ECCV 2012. Congratulations Philipp!

Philipp Krähenbühl won the Outstanding Student Paper award at NIPS 2011 for our paper Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials. This year NIPS received 1,400 submissions, 306 of which were accepted for publication. Three of these were picked for the award. (See the conference book for more details.)

Paper published in Science Robotics and featured on the cover

Paper published in Nature and featured on the cover

SIGGRAPH Test-of-Time Award

Paper accepted to SIGGRAPH 2023

Two papers accepted to PAMI

Four papers accepted to NeurIPS 2022

Enhancing Photorealism Enhancement accepted to PAMI

Four papers accepted to CVPR 2022

Two papers accepted to ICLR 2022

Paper published in Science Robotics

Paper published in Science Robotics

Three papers accepted to NeurIPS 2021

Five papers accepted to ICCV 2021

Paper published in PLOS ONE

Four papers accepted to ICML 2021

Two papers accepted to CVPR 2021

Paper accepted to ICRA 2021

Paper accepted to ICLR 2021

Paper published in Science Robotics and featured on the cover

Paper accepted to NeurIPS 2020

Three papers accepted to ECCV 2020

“Deep Drone Acrobatics” nominated for Best Paper Award

Two papers accepted to ICML 2020

Paper accepted to RSS 2020

Five papers accepted to CVPR 2020

Three papers accepted to ICLR 2020

“Habitat” nominated for Best Paper Award

Paper accepted to CoRL 2019

Two papers accepted to NeurIPS 2019

Five papers accepted to ICCV 2019

Paper accepted to IEEE Robotics and Automation Letters

Paper accepted to Interspeech 2019

Paper published in Science Robotics

Paper accepted to SIGGRAPH 2019

Five papers accepted to CVPR 2019

Paper accepted to ICRA 2019

Paper published in Science Robotics

Two papers accepted to ICLR 2019

CoRL Best Systems Paper Award

Two papers accepted to NIPS 2018

Three papers accepted to CoRL 2018

Two papers accepted to ECCV 2018

Paper accepted to IEEE Robotics and Automation Letters

Four papers accepted to CVPR 2018

Open3D released

Two papers accepted to ICLR 2018

Paper accepted to ICRA 2018

Papers accepted to NIPS and CoRL

Paper published in PNAS

Five papers accepted to ICCV 2017

Paper accepted to SIGGRAPH 2017

Two papers accepted to CVPR 2017

Paper accepted to ICLR 2017

Paper accepted to PAMI

Visual Doom AI Competition

Two papers accepted to ECCV 2016

Three papers accepted to CVPR 2016

Paper accepted to ICLR 2016

Adobe releases Fuse

Paper accepted to ICCV 2015

Paper accepted to SIGGRAPH 2015

Three papers accepted to CVPR 2015

Joined Intel Labs

Paper accepted to ECCV 2014

Paper accepted to SIGGRAPH 2014

Paper accepted to ICML 2014

Two papers accepted to CVPR 2014

Symposium on Computer Animation 2014

Paper accepted to NIPS 2013

Two papers accepted to ICCV 2013

Paper accepted to SIGGRAPH Asia 2013

Two papers accepted to ICML 2013

Paper accepted to SIGGRAPH 2013

Mixamo releases Fuse

Paper accepted to ECCV 2012

Paper accepted to ICML 2012

Four papers accepted to SIGGRAPH 2012

NIPS Outstanding Student Paper Award