Category Archives: Uncategorized

Four papers were accepted to the International Conference on Learning Representations (ICLR): Does Spatial Cognition Emerge in Frontier Models?, Depth Pro: Sharp Monocular Metric Depth in Less Than a Second, CoMotion: Concurrent Multi-person 3D Motion, and Cut Your Losses in Large-Vocabulary Language Models. Cut Your Losses was selected for oral presentation at the conference (1.8% acceptance rate).

Reaching the Limit in Autonomous Racing: Optimal Control versus Reinforcement Learning was published in Science Robotics and featured on the journal’s cover. The paper presents a systematic study of control system design methodologies, focusing on reinforcement learning and optimal control in the context of autonomous drone racing. We show that neural networks trained with reinforcement learning (RL) outperform optimal control methods and trace the root cause to the flexibility afforded by RL in the formulation of the controller’s objective. In conjunction with the study, we push autonomous drones to unprecedented performance regimes, demonstrating superhuman control while reaching accelerations greater than 12g and velocities greater than 100 km/h.

Champion-level Drone Racing using Deep Reinforcement Learning was published in Nature and featured on the journal’s cover. The paper presents an autonomous system that outraced human world champions in first-person view (FPV) drone racing, a televised sport. This is the first time that an autonomous mobile robot achieved world-champion-level performance in a real-world competitive physical sport. Here is a video created by Nature to highlight the work and here is an accompanying article that provides a good overview.

The paper Optimizing Locomotion Controllers Using Biologically-Based Actuators and Objectives received the ACM SIGGRAPH Test-of-Time Award at SIGGRAPH 2023. The award “recognizes highly influential papers published in SIGGRAPH conferences that have made a significant impact over the past 10 years or more”. This is the first year of this annual award, for which all papers presented at SIGGRAPH conferences from 2011 to 2013 were considered. Four papers were selected for the award.

The paper An Extensible, Data-Oriented Architecture for High-Performance, Many-World Simulation was accepted to SIGGRAPH 2023.

Two papers were accepted to the IEEE Transactions on Pattern Analysis and Machine Intelligence: Drinking from a Firehose: Continual Learning with Web-scale Natural Language and ASH: A Modern Framework for Parallel Spatial Hashing in 3D Perception.

Four papers were accepted to Neural Information Processing Systems (NeurIPS): Non-deep Networks, Domain Generalization without Excess Empirical Risk, Guaranteed Conservation of Momentum for Learning Particle-based Fluid Dynamics, and Scale-invariant Learning by Physics Inversion. Guaranteed Conservation of Momentum for Learning Particle-based Fluid Dynamics was selected for oral presentation at the conference.

The paper Enhancing Photorealism Enhancement was accepted to the IEEE Transactions on Pattern Analysis and Machine Intelligence.

Four papers were accepted to the Conference on Computer Vision and Pattern Recognition (CVPR): Dancing under the stars: video denoising in starlight, Simple Multi-dataset Detection, Global Tracking Transformers, and Shape from Polarization for Complex Scenes in the Wild. Dancing under the stars was selected for oral presentation at the conference.

Two papers were accepted to the International Conference on Learning Representations (ICLR): Language-driven Semantic Segmentation and Neural Deep Equilibrium Solvers.

Learning robust perceptive locomotion for quadrupedal robots in the wild was published in Science Robotics. We have combined proprioception with visual perception to create a locomotion controller with unprecedented versatility and robustness. The controller went on an hour-long hike up and down a mountain in Switzerland, without a single fall, completing the trail in the same time as recommended in the hiking guide for humans. It also powered all the legged robots for the winning team in the DARPA Subterranean Challenge. The key contribution is combining visual perception with proprioception. Everything is trained purely in simulation and transferred zero-shot to reality.

Learning high-speed flight in the wild was published in Science Robotics. The paper presents an end-to-end approach to autonomous high-speed flight through complex natural and human-made environments, with purely onboard sensing and computation. The flight is controlled by a convolutional network that maps noisy sensory observations to collision-free trajectories in a receding-horizon fashion. It is trained exclusively in simulation via privileged learning.

Three papers were accepted to Neural Information Processing Systems (NeurIPS): Geometry Processing with Neural Fields, Differentiable Simulation of Soft Multi-body Systems, and Habitat 2.0: Training Home Assistants to Rearrange their Habitat. Habitat 2.0 was selected for a spotlight at the conference (<3% acceptance rate).

Five papers were accepted to the International Conference on Computer Vision (ICCV). Two of these were selected for full oral presentation at the conference (3.4% acceptance rate): Point Transformer and Learning to Drive from a World on Rails.

The h-index is no longer an effective correlate of scientific reputation was published in PLOS ONE.

Four papers were accepted to the International Conference on Machine Learning (ICML): Megaverse: Simulating Embodied Agents at One Million Experiences per Second, Efficient Differentiable Simulation of Articulated Bodies, Stabilizing Equilibrium Models by Jacobian Regularization, and Training Graph Neural Networks with 1000 Layers.

Two papers were accepted to the Conference on Computer Vision and Pattern Recognition (CVPR): Self-supervised Geometric Perception and Stable View Synthesis. Self-supervised Geometric Perception was selected for oral presentation at the conference.

The paper OpenBot: Turning Smartphones into Robots was accepted to the International Conference on Robotics and Automation (ICRA).

The paper Large Batch Simulation for Deep Reinforcement Learning was accepted to the International Conference on Learning Representations (ICLR).

Learning Quadrupedal Locomotion over Challenging Terrain was published in Science Robotics and featured on the journal’s cover. The paper presents a radically robust legged locomotion controller for rough terrain and demonstrates remarkable zero-shot generalization from simulation to natural environments. The controller retains its robustness under conditions that have never been encountered during training: deformable terrain such as mud and snow, dynamic footholds such as rubble, and overground impediments such as thick vegetation and gushing water.

The paper Multiscale Deep Equilibrium Models was accepted to Neural Information Processing Systems (NeurIPS) and selected for oral presentation at the conference (1.1% acceptance rate).

Three papers were accepted to the European Conference on Computer Vision (ECCV): Free View Synthesis, Tracking Objects as Points, and Dynamic Low-light Imaging with Quanta Image Sensors. Tracking Objects as Points was selected for a spotlight presentation at the conference (5.3% acceptance rate).

Deep Drone Acrobatics was nominated for the Best Paper Award at the Robotics: Science and Systems (RSS) 2020 conference. It was one of 3 papers nominated for the award, out of 321 submissions.

Two papers were accepted to the International Conference on Machine Learning (ICML): Sample Factory and Scalable Differentiable Physics for Learning and Control.

The paper Deep Drone Acrobatics was accepted to Robotics: Science and Systems (RSS).

Five papers were accepted to the Conference on Computer Vision and Pattern Recognition (CVPR). Deep Global Registration and High-dimensional Convolutional Networks were selected for oral presentation at the conference (5.7% acceptance rate).

Three papers were accepted to the International Conference on Learning Representations (ICLR): Learning to Guide Random Search, Learning to Control PDEs with Differentiable Physics, and Lagrangian Fluid Simulation with Continuous Convolutions. Learning to Control PDEs was selected for a spotlight presentation at the conference (6% acceptance rate).

Habitat: A Platform for Embodied AI Research was nominated for the Best Paper Award at the International Conference on Computer Vision (ICCV). It was one of 11 papers nominated or selected for awards at the conference, out of 4303 submissions.

The paper Learning by Cheating was accepted to the Conference on Robot Learning (CoRL).

Two papers were accepted to Neural Information Processing Systems (NeurIPS): Deep Equilibrium Models and Differentiable Cloth Simulation for Inverse Problems. Deep Equilibrium Models were selected for a spotlight presentation at the conference (3% acceptance rate).

Five papers were accepted to the International Conference on Computer Vision (ICCV). Three of these were selected for full oral presentation at the conference (4.6% acceptance rate): Seeing Motion in the Dark, Consensus Maximization Tree Search Revisited, and Habitat: A Platform for Embodied AI Research

The paper Trajectory Optimization for Legged Robots With Slipping Motions was accepted to IEEE Robotics and Automation Letters.

The paper Speech Denoising with Deep Feature Losses was accepted to Interspeech 2019.

Does Computer Vision Matter for Action? was published in Science Robotics. The paper studies whether explicit intermediate representations of the kind studied in computer vision help action. We probe this question via controlled experiments in immersive simulation. Our main finding is that computer vision does matter. Models equipped with intermediate representations train faster, achieve higher task performance, and generalize better to previously unseen environments.

The paper A Learned Shape-Adaptive Subsurface Scattering Model was accepted to SIGGRAPH 2019.

Five papers were accepted to the Conference on Computer Vision and Pattern Recognition (CVPR). Acoustic Non-Line-of-Sight Imaging was selected for oral presentation at the conference (5.6% acceptance rate).

The paper Beauty and the Beast: Optimal Methods Meet Learning for Drone Racing was accepted to the International Conference on Robotics and Automation (ICRA).

Learning Agile and Dynamic Motor Skills for Legged Robots was published in Science Robotics. The paper introduces a method for training a neural network policy in simulation and transferring it to a state-of-the-art legged system. The approach is applied to the ANYmal robot, a sophisticated medium-dog–sized quadrupedal machine. Using policies trained in simulation, the robot achieves locomotion skills that go beyond what had been achieved with prior methods: ANYmal is capable of precisely and energy-efficiently following high-level body velocity commands, running faster than before, and recovering from falling even in complex configurations.

Two papers were accepted to the International Conference on Learning Representations (ICLR): Trellis Networks for Sequence Modeling and Deep Layers as Stochastic Solvers.

Our paper Deep Drone Racing: Learning Agile Flight in Dynamic Environments received the Best Systems Paper Award at the Conference on Robot Learning (CoRL). Three papers were selected for awards of any kind, out of 237 submissions to the conference.

Two papers were accepted to Neural Information Processing Systems (NIPS): Multi-Task Learning as Multi-Objective Optimization and Combinatorial Optimization with Graph Convolutional Networks and Guided Tree Search.

Three papers were accepted to the Conference on Robot Learning (CoRL): Deep Drone Racing, Driving Policy Transfer, and Motion Perception in Reinforcement Learning with Dynamic Objects. Deep Drone Racing was selected for full oral presentation at the conference.

Two papers were accepted to the European Conference on Computer Vision (ECCV): Deep Fundamental Matrix Estimation and On Offline Evaluation of Vision-based Driving Models.

The paper Trajectory Optimization with Implicit Hard Contacts was accepted to IEEE Robotics and Automation Letters.

Four papers were accepted to the Conference on Computer Vision and Pattern Recognition (CVPR). Semi-parametric Image Synthesis was selected for full oral presentation at the conference (2.1% acceptance rate). Tangent Convolutions for Dense Prediction in 3D was selected for a spotlight oral.

We have released Open3D, a modern open-source library for 3D data processing. Open3D has been in development and internal use for three years, has supported multiple published research projects, and is actively deployed in the cloud. It is released open-source under the permissive MIT license and is now available at www.open3d.org.

Two papers were accepted to the International Conference on Learning Representations (ICLR): TD or not TD and Semi-parametric Topological Memory for Navigation.

The paper End-to-end Driving via Conditional Imitation Learning was accepted to the International Conference on Robotics and Automation (ICRA).

Two new papers accepted. Learning to Inpaint for Image Compression was accepted to Neural Information Processing Systems (NIPS). And CARLA: An Open Urban Driving Simulator was accepted to the new Conference on Robot Learning (CoRL).

Robust Continuous Clustering was published in the Proceedings of the National Academy of Sciences (PNAS). The paper presents a clustering algorithm that optimizes a smooth global objective using efficient numerical methods. This allows clustering to be integrated into end-to-end feature learning pipelines. Our algorithm effectively untangles mixed clusters, achieves high accuracy across domains, and scales to high dimensions and large datasets.

Five papers were accepted to the International Conference on Computer Vision (ICCV). Photographic Image Synthesis was selected for full oral presentation at the conference (2.1% acceptance rate). Playing for Benchmarks was selected for a spotlight oral.

The paper Tanks and Temples: Benchmarking Large-Scale Scene Reconstruction was accepted to SIGGRAPH 2017.

Two papers were accepted to the Conference on Computer Vision and Pattern Recognition (CVPR): Dilated Residual Networks and Accurate Optical Flow via Direct Cost Volume Processing.

The paper Learning to Act by Predicting the Future was accepted to the International Conference on Learning Representations (ICLR) and selected for full oral presentation at the conference (3% acceptance rate).

The paper Direct Sparse Odometry was accepted to the IEEE Transactions on Pattern Analysis and Machine Intelligence.

Our entry won the Visual Doom AI Competition. The competition evaluates sensorimotor control models that act in three-dimensional environments based on raw sensory input. Our entry, IntelAct, placed first in the “Full deathmatch” track, which takes place in previously unseen environments. Our model was trained end-to-end: from pixels to actions. It learns to act from raw experience, without human gameplay or other “expert” supervision. The work is based on an approach to sensorimotor control that will be presented in a forthcoming research paper. A video preview is available.

Two papers were accepted to the European Conference on Computer Vision (ECCV): Playing for Data and Fast Global Registration. The latter was selected for full oral presentation at the conference (1.8% acceptance rate).

Three papers were accepted to the Conference on Computer Vision and Pattern Recognition (CVPR). Two of these were selected for full oral presentation at the conference (3.9% acceptance rate).

The paper Multi-Scale Context Aggregation by Dilated Convolutions was accepted to the International Conference on Learning Representations (ICLR).

Adobe released a product based on our data-driven 3D modeling research at Stanford. The original research was described in papers published at SIGGRAPH Asia 2010 and SIGGRAPH 2011, and was the subject of Siddhartha Chaudhuri’s PhD thesis. Our technology was licensed in 2012 by Mixamo, a start-up company led by Stefano Corazza. Mixamo turned the technology into a product called Fuse. Fuse became a widely used 3D character modeling tool and led to Adobe’s acquisition of Mixamo earlier this year. Fuse has now been rebranded and launched as part of the Adobe Creative Cloud.

The paper Robust Nonrigid Registration by Convex Optimization was accepted to the International Conference on Computer Vision (ICCV) and selected for full oral presentation at the conference (3.3% acceptance rate).

The paper Single-View Reconstruction via Joint Analysis of Image and Shape Collections was accepted to SIGGRAPH 2015.

Three papers were accepted to the Conference on Computer Vision and Pattern Recognition (CVPR).

Joined Intel Labs to build a new lab devoted to basic research.

The paper Geodesic Object Proposals was accepted to the European Conference on Computer Vision (ECCV) and selected for full oral presentation at the conference (2.8% acceptance rate).

The paper Color Map Optimization for 3D Reconstruction with Consumer Depth Cameras was accepted to SIGGRAPH 2014.

A new paper on policy search was accepted to the International Conference on Machine Learning (ICML).

Two papers were accepted to the Conference on Computer Vision and Pattern Recognition (CVPR): Fast MRF Optimization with Application to Depth Reconstruction and Simultaneous Localization and Calibration.

I’m serving as a Program Chair of the Symposium on Computer Animation. The submission deadline is April 15. The call for papers is here.

A new paper on policy search was accepted to NIPS 2013.

Two papers were accepted to the International Conference on Computer Vision (ICCV): A Simple Model for Intrinsic Image Decomposition with Depth Cues and Elastic Fragments for Dense Scene Reconstruction. The latter was selected for full oral presentation at the conference (2.5% acceptance rate).

A new paper on simulation of human motion was accepted to SIGGRAPH Asia 2013.

The papers Guided Policy Search and Parameter Learning and Convergent Inference for Dense Random Fields were accepted to ICML 2013.

The paper Dense Scene Reconstruction with Points of Interest was accepted to SIGGRAPH 2013.

Mixamo released a character modeling tool based on our assembly-based modeling work. The modeling tool is called Fuse and is a polished version of the system developed by Siddhartha Chaudhuri for his PhD thesis.

The paper Efficient Nonlocal Regularization for Optical Flow was accepted to ECCV 2012. Congratulations Philipp!

The paper Continuous Inverse Optimal Control with Locally Optimal Examples was accepted to ICML 2012. Congratulations Sergey!

Four papers were accepted to SIGGRAPH 2012: A Probabilistic Model for Component-Based Shape Synthesis, Optimizing Locomotion Controllers Using Biologically-Based Actuators and Objectives, An Algebraic Model for Parameterized Shape Editing, and Continuous Character Control with Low-Dimensional Embeddings.

Philipp Krähenbühl won the Outstanding Student Paper award at NIPS 2011 for our paper Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials. This year NIPS received 1,400 submissions, 306 of which were accepted for publication. Three of these were picked for the award. (See the conference book for more details.)