News

Paper published in Science Robotics and featured on the cover


Posted on October 10, 2023

Reaching the Limit in Autonomous Racing: Optimal Control versus Reinforcement Learning was published in Science Robotics and featured on the journal’s cover. The paper presents a systematic study of control system design methodologies, focusing on reinforcement learning and optimal control in the context of autonomous drone racing. We show that neural networks trained with reinforcement learning (RL) outperform optimal control methods and trace the root cause to the flexibility afforded by RL in the formulation of the controller’s objective. In conjunction with the study, we push autonomous drones to unprecedented performance regimes, demonstrating superhuman control while reaching accelerations greater than 12g and velocities greater than 100 km/h.

Paper published in Nature and featured on the cover


Posted on October 10, 2023

Champion-level Drone Racing using Deep Reinforcement Learning was published in Nature and featured on the journal’s cover. The paper presents an autonomous system that outraced human world champions in first-person view (FPV) drone racing, a televised sport. This is the first time that an autonomous mobile robot achieved world-champion-level performance in a real-world competitive physical sport. Here is a video created by Nature to highlight the work and here is an accompanying article that provides a good overview.

SIGGRAPH Test-of-Time Award


Posted on August 12, 2023

The paper Optimizing Locomotion Controllers Using Biologically-Based Actuators and Objectives received the ACM SIGGRAPH Test-of-Time Award at SIGGRAPH 2023. The award “recognizes highly influential papers published in SIGGRAPH conferences that have made a significant impact over the past 10 years or more”. This is the first year of this annual award, for which all papers presented at SIGGRAPH conferences from 2011 to 2013 were considered. Four papers were selected for the award.

Paper accepted to SIGGRAPH 2023


Posted on August 12, 2023

The paper An Extensible, Data-Oriented Architecture for High-Performance, Many-World Simulation was accepted to SIGGRAPH 2023.

Enhancing Photorealism Enhancement accepted to PAMI


Posted on March 21, 2022

The paper Enhancing Photorealism Enhancement was accepted to the IEEE Transactions on Pattern Analysis and Machine Intelligence.

Two papers accepted to ICLR 2022


Posted on February 13, 2022

Two papers were accepted to the International Conference on Learning Representations (ICLR): Language-driven Semantic Segmentation and Neural Deep Equilibrium Solvers.

Paper published in Science Robotics


Posted on February 13, 2022

Learning robust perceptive locomotion for quadrupedal robots in the wild was published in Science Robotics. We have combined proprioception with visual perception to create a locomotion controller with unprecedented versatility and robustness. The controller went on an hour-long hike up and down a mountain in Switzerland, without a single fall, completing the trail in the same time as recommended in the hiking guide for humans. It also powered all the legged robots for the winning team in the DARPA Subterranean Challenge. The key contribution is combining visual perception with proprioception. Everything is trained purely in simulation and transferred zero-shot to reality.

Paper published in Science Robotics


Posted on October 7, 2021

Learning high-speed flight in the wild was published in Science Robotics. The paper presents an end-to-end approach to autonomous high-speed flight through complex natural and human-made environments, with purely onboard sensing and computation. The flight is controlled by a convolutional network that maps noisy sensory observations to collision-free trajectories in a receding-horizon fashion. It is trained exclusively in simulation via privileged learning.

Three papers accepted to NeurIPS 2021


Posted on October 3, 2021

Three papers were accepted to Neural Information Processing Systems (NeurIPS): Geometry Processing with Neural Fields, Differentiable Simulation of Soft Multi-body Systems, and Habitat 2.0: Training Home Assistants to Rearrange their Habitat. Habitat 2.0 was selected for a spotlight at the conference (<3% acceptance rate).

Five papers accepted to ICCV 2021


Posted on July 25, 2021

Five papers were accepted to the International Conference on Computer Vision (ICCV). Two of these were selected for full oral presentation at the conference (3.4% acceptance rate): Point Transformer and Learning to Drive from a World on Rails.

Paper published in PLOS ONE


Posted on June 29, 2021

The h-index is no longer an effective correlate of scientific reputation was published in PLOS ONE.

Two papers accepted to CVPR 2021


Posted on March 7, 2021

Two papers were accepted to the Conference on Computer Vision and Pattern Recognition (CVPR): Self-supervised Geometric Perception and Stable View Synthesis. Self-supervised Geometric Perception was selected for oral presentation at the conference.

Paper accepted to ICRA 2021


Posted on March 7, 2021

The paper OpenBot: Turning Smartphones into Robots was accepted to the International Conference on Robotics and Automation (ICRA).

Paper accepted to ICLR 2021


Posted on January 14, 2021

The paper Large Batch Simulation for Deep Reinforcement Learning was accepted to the International Conference on Learning Representations (ICLR).

Paper published in Science Robotics and featured on the cover


Posted on October 22, 2020

Learning Quadrupedal Locomotion over Challenging Terrain was published in Science Robotics and featured on the journal’s cover. The paper presents a radically robust legged locomotion controller for rough terrain and demonstrates remarkable zero-shot generalization from simulation to natural environments. The controller retains its robustness under conditions that have never been encountered during training: deformable terrain such as mud and snow, dynamic footholds such as rubble, and overground impediments such as thick vegetation and gushing water.

Paper accepted to NeurIPS 2020


Posted on September 26, 2020

The paper Multiscale Deep Equilibrium Models was accepted to Neural Information Processing Systems (NeurIPS) and selected for oral presentation at the conference (1.1% acceptance rate).

Free View Synthesis

Three papers accepted to ECCV 2020


Posted on July 18, 2020

Three papers were accepted to the European Conference on Computer Vision (ECCV): Free View Synthesis, Tracking Objects as Points, and Dynamic Low-light Imaging with Quanta Image Sensors. Tracking Objects as Points was selected for a spotlight presentation at the conference (5.3% acceptance rate).

“Deep Drone Acrobatics” nominated for Best Paper Award


Posted on July 18, 2020

Deep Drone Acrobatics was nominated for the Best Paper Award at the Robotics: Science and Systems (RSS) 2020 conference. It was one of 3 papers nominated for the award, out of 321 submissions.

Two papers accepted to ICML 2020


Posted on June 11, 2020

Two papers were accepted to the International Conference on Machine Learning (ICML): Sample Factory and Scalable Differentiable Physics for Learning and Control.

Paper accepted to RSS 2020


Posted on June 11, 2020

The paper Deep Drone Acrobatics was accepted to Robotics: Science and Systems (RSS).

Five papers accepted to CVPR 2020


Posted on April 4, 2020

Five papers were accepted to the Conference on Computer Vision and Pattern Recognition (CVPR). Deep Global Registration and High-dimensional Convolutional Networks were selected for oral presentation at the conference (5.7% acceptance rate).

Three papers accepted to ICLR 2020


Posted on January 11, 2020

Three papers were accepted to the International Conference on Learning Representations (ICLR): Learning to Guide Random Search, Learning to Control PDEs with Differentiable Physics, and Lagrangian Fluid Simulation with Continuous Convolutions. Learning to Control PDEs was selected for a spotlight presentation at the conference (6% acceptance rate).

“Habitat” nominated for Best Paper Award


Posted on October 31, 2019

Habitat: A Platform for Embodied AI Research was nominated for the Best Paper Award at the International Conference on Computer Vision (ICCV). It was one of 11 papers nominated or selected for awards at the conference, out of 4303 submissions.

Paper accepted to CoRL 2019


Posted on September 15, 2019

The paper Learning by Cheating was accepted to the Conference on Robot Learning (CoRL).

Two papers accepted to NeurIPS 2019


Posted on September 4, 2019

Two papers were accepted to Neural Information Processing Systems (NeurIPS): Deep Equilibrium Models and Differentiable Cloth Simulation for Inverse Problems. Deep Equilibrium Models were selected for a spotlight presentation at the conference (3% acceptance rate).

Five papers accepted to ICCV 2019


Posted on August 3, 2019

Five papers were accepted to the International Conference on Computer Vision (ICCV). Three of these were selected for full oral presentation at the conference (4.6% acceptance rate): Seeing Motion in the Dark, Consensus Maximization Tree Search Revisited, and Habitat: A Platform for Embodied AI Research

Paper accepted to IEEE Robotics and Automation Letters


Posted on August 3, 2019

The paper Trajectory Optimization for Legged Robots With Slipping Motions was accepted to IEEE Robotics and Automation Letters.

Paper accepted to Interspeech 2019


Posted on August 3, 2019

The paper Speech Denoising with Deep Feature Losses was accepted to Interspeech 2019.

Paper published in Science Robotics


Posted on May 23, 2019

Does Computer Vision Matter for Action? was published in Science Robotics. The paper studies whether explicit intermediate representations of the kind studied in computer vision help action. We probe this question via controlled experiments in immersive simulation. Our main finding is that computer vision does matter. Models equipped with intermediate representations train faster, achieve higher task performance, and generalize better to previously unseen environments.

Paper accepted to SIGGRAPH 2019


Posted on May 5, 2019

The paper A Learned Shape-Adaptive Subsurface Scattering Model was accepted to SIGGRAPH 2019.

Five papers accepted to CVPR 2019


Posted on March 9, 2019

Five papers were accepted to the Conference on Computer Vision and Pattern Recognition (CVPR). Acoustic Non-Line-of-Sight Imaging was selected for oral presentation at the conference (5.6% acceptance rate).

Paper accepted to ICRA 2019


Posted on February 3, 2019

The paper Beauty and the Beast: Optimal Methods Meet Learning for Drone Racing was accepted to the International Conference on Robotics and Automation (ICRA).

Paper published in Science Robotics


Posted on January 20, 2019

Learning Agile and Dynamic Motor Skills for Legged Robots was published in Science Robotics. The paper introduces a method for training a neural network policy in simulation and transferring it to a state-of-the-art legged system. The approach is applied to the ANYmal robot, a sophisticated medium-dog–sized quadrupedal machine. Using policies trained in simulation, the robot achieves locomotion skills that go beyond what had been achieved with prior methods: ANYmal is capable of precisely and energy-efficiently following high-level body velocity commands, running faster than before, and recovering from falling even in complex configurations.

Two papers accepted to ICLR 2019


Posted on December 27, 2018

Two papers were accepted to the International Conference on Learning Representations (ICLR): Trellis Networks for Sequence Modeling and Deep Layers as Stochastic Solvers.

CoRL Best Systems Paper Award


Posted on November 1, 2018

Our paper Deep Drone Racing: Learning Agile Flight in Dynamic Environments received the Best Systems Paper Award at the Conference on Robot Learning (CoRL). Three papers were selected for awards of any kind, out of 237 submissions to the conference.

Three papers accepted to CoRL 2018


Posted on September 7, 2018

Three papers were accepted to the Conference on Robot Learning (CoRL): Deep Drone Racing, Driving Policy Transfer, and Motion Perception in Reinforcement Learning with Dynamic Objects. Deep Drone Racing was selected for full oral presentation at the conference.

Two papers accepted to ECCV 2018


Posted on August 16, 2018

Two papers were accepted to the European Conference on Computer Vision (ECCV): Deep Fundamental Matrix Estimation and On Offline Evaluation of Vision-based Driving Models.

Paper accepted to IEEE Robotics and Automation Letters


Posted on August 16, 2018

The paper Trajectory Optimization with Implicit Hard Contacts was accepted to IEEE Robotics and Automation Letters.

Four papers accepted to CVPR 2018


Posted on March 4, 2018

Four papers were accepted to the Conference on Computer Vision and Pattern Recognition (CVPR). Semi-parametric Image Synthesis was selected for full oral presentation at the conference (2.1% acceptance rate). Tangent Convolutions for Dense Prediction in 3D was selected for a spotlight oral.

Open3D released


Posted on January 31, 2018

We have released Open3D, a modern open-source library for 3D data processing. Open3D has been in development and internal use for three years, has supported multiple published research projects, and is actively deployed in the cloud. It is released open-source under the permissive MIT license and is now available at www.open3d.org.

Two papers accepted to ICLR 2018


Posted on January 31, 2018

Two papers were accepted to the International Conference on Learning Representations (ICLR): TD or not TD and Semi-parametric Topological Memory for Navigation.

Paper accepted to ICRA 2018


Posted on January 20, 2018

The paper End-to-end Driving via Conditional Imitation Learning was accepted to the International Conference on Robotics and Automation (ICRA).

Papers accepted to NIPS and CoRL


Posted on September 9, 2017

Two new papers accepted. Learning to Inpaint for Image Compression was accepted to Neural Information Processing Systems (NIPS). And CARLA: An Open Urban Driving Simulator was accepted to the new Conference on Robot Learning (CoRL).

Paper published in PNAS


Posted on August 30, 2017

Robust Continuous Clustering was published in the Proceedings of the National Academy of Sciences (PNAS). The paper presents a clustering algorithm that optimizes a smooth global objective using efficient numerical methods. This allows clustering to be integrated into end-to-end feature learning pipelines. Our algorithm effectively untangles mixed clusters, achieves high accuracy across domains, and scales to high dimensions and large datasets.

Playing for Benchmarks

Five papers accepted to ICCV 2017


Posted on July 31, 2017

Five papers were accepted to the International Conference on Computer Vision (ICCV). Photographic Image Synthesis was selected for full oral presentation at the conference (2.1% acceptance rate). Playing for Benchmarks was selected for a spotlight oral.

Paper accepted to SIGGRAPH 2017


Posted on March 27, 2017

The paper Tanks and Temples: Benchmarking Large-Scale Scene Reconstruction was accepted to SIGGRAPH 2017.

Two papers accepted to CVPR 2017


Posted on March 4, 2017

Two papers were accepted to the Conference on Computer Vision and Pattern Recognition (CVPR): Dilated Residual Networks and Accurate Optical Flow via Direct Cost Volume Processing.

Paper accepted to ICLR 2017


Posted on February 7, 2017

The paper Learning to Act by Predicting the Future was accepted to the International Conference on Learning Representations (ICLR) and selected for full oral presentation at the conference (3% acceptance rate).

Paper accepted to PAMI


Posted on February 7, 2017

The paper Direct Sparse Odometry was accepted to the IEEE Transactions on Pattern Analysis and Machine Intelligence.

Visual Doom AI Competition


Posted on September 26, 2016

Our entry won the Visual Doom AI Competition. The competition evaluates sensorimotor control models that act in three-dimensional environments based on raw sensory input. Our entry, IntelAct, placed first in the “Full deathmatch” track, which takes place in previously unseen environments. Our model was trained end-to-end: from pixels to actions. It learns to act from raw experience, without human gameplay or other “expert” supervision. The work is based on an approach to sensorimotor control that will be presented in a forthcoming research paper. A video preview is available.

Two papers accepted to ECCV 2016


Posted on August 1, 2016

Two papers were accepted to the European Conference on Computer Vision (ECCV): Playing for Data and Fast Global Registration. The latter was selected for full oral presentation at the conference (1.8% acceptance rate).

Three papers accepted to CVPR 2016


Posted on March 4, 2016

Three papers were accepted to the Conference on Computer Vision and Pattern Recognition (CVPR). Two of these were selected for full oral presentation at the conference (3.9% acceptance rate).

Paper accepted to ICLR 2016


Posted on March 4, 2016

The paper Multi-Scale Context Aggregation by Dilated Convolutions was accepted to the International Conference on Learning Representations (ICLR).

Adobe Fuse

Adobe releases Fuse


Posted on October 10, 2015

Adobe released a product based on our data-driven 3D modeling research at Stanford. The original research was described in papers published at SIGGRAPH Asia 2010 and SIGGRAPH 2011, and was the subject of Siddhartha Chaudhuri’s PhD thesis. Our technology was licensed in 2012 by Mixamo, a start-up company led by Stefano Corazza. Mixamo turned the technology into a product called Fuse. Fuse became a widely used 3D character modeling tool and led to Adobe’s acquisition of Mixamo earlier this year. Fuse has now been rebranded and launched as part of the Adobe Creative Cloud.

Robust Nonrigid Registration by Convex Optimization

Paper accepted to ICCV 2015


Posted on September 6, 2015

The paper Robust Nonrigid Registration by Convex Optimization was accepted to the International Conference on Computer Vision (ICCV) and selected for full oral presentation at the conference (3.3% acceptance rate).

http://vladlen.info/publications/single-view-reconstruction-via-joint-analysis-of-image-and-shape-collections/

Paper accepted to SIGGRAPH 2015


Posted on May 8, 2015

The paper Single-View Reconstruction via Joint Analysis of Image and Shape Collections was accepted to SIGGRAPH 2015.

Three papers accepted to CVPR 2015


Posted on March 9, 2015

Three papers were accepted to the Conference on Computer Vision and Pattern Recognition (CVPR).

Joined Intel Labs


Posted on January 5, 2015

Joined Intel Labs to build a new lab devoted to basic research.

Geodesic Object Proposals

Paper accepted to ECCV 2014


Posted on July 8, 2014

The paper Geodesic Object Proposals was accepted to the European Conference on Computer Vision (ECCV) and selected for full oral presentation at the conference (2.8% acceptance rate).

Paper accepted to SIGGRAPH 2014


Posted on April 24, 2014

The paper Color Map Optimization for 3D Reconstruction with Consumer Depth Cameras was accepted to SIGGRAPH 2014.

Paper accepted to ICML 2014


Posted on April 24, 2014

A new paper on policy search was accepted to the International Conference on Machine Learning (ICML).

Symposium on Computer Animation 2014

Symposium on Computer Animation 2014


Posted on February 23, 2014

I’m serving as a Program Chair of the Symposium on Computer Animation. The submission deadline is April 15. The call for papers is here.

Paper accepted to NIPS 2013


Posted on October 8, 2013

A new paper on policy search was accepted to NIPS 2013.

Two papers accepted to ICCV 2013


Posted on September 2, 2013

Two papers were accepted to the International Conference on Computer Vision (ICCV): A Simple Model for Intrinsic Image Decomposition with Depth Cues and Elastic Fragments for Dense Scene Reconstruction. The latter was selected for full oral presentation at the conference (2.5% acceptance rate).

Paper accepted to SIGGRAPH Asia 2013


Posted on July 24, 2013

A new paper on simulation of human motion was accepted to SIGGRAPH Asia 2013.

Two papers accepted to ICML 2013


Posted on May 4, 2013

The papers Guided Policy Search and Parameter Learning and Convergent Inference for Dense Random Fields were accepted to ICML 2013.

Paper accepted to SIGGRAPH 2013


Posted on April 21, 2013

The paper Dense Scene Reconstruction with Points of Interest was accepted to SIGGRAPH 2013.

Mixamo releases Fuse

Mixamo releases Fuse


Posted on August 27, 2012

Mixamo released a character modeling tool based on our assembly-based modeling work. The modeling tool is called Fuse and is a polished version of the system developed by Siddhartha Chaudhuri for his PhD thesis.

Paper accepted to ECCV 2012


Posted on June 27, 2012

The paper Efficient Nonlocal Regularization for Optical Flow was accepted to ECCV 2012. Congratulations Philipp!

Continuous Inverse Optimal Control with Locally Optimal Examples

Paper accepted to ICML 2012


Posted on May 15, 2012

The paper Continuous Inverse Optimal Control with Locally Optimal Examples was accepted to ICML 2012. Congratulations Sergey!

NIPS Outstanding Student Paper Award


Posted on December 9, 2011

Philipp Krähenbühl won the Outstanding Student Paper award at NIPS 2011 for our paper Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials. This year NIPS received 1,400 submissions, 306 of which were accepted for publication. Three of these were picked for the award. (See the conference book for more details.)