Sensorimotor Control and Simulation

Project Overview

We are working on sensorimotor control: learning to act based on raw sensory input. Our models and algorithms aim to support flexible operation in complex and dynamic three-dimensional environments. We are inspired by applications such as autonomous driving and household robotics, and by scientific curiosity. Much of our work leverages immersive simulation, and we have developed simulation platforms to support this field.

Publications

An Extensible, Data-Oriented Architecture for High-Performance, Many-World Simulation

Brennan Shacklett, Luc Guy Rosenzweig, Zhiqiang Xie, Bidipta Sarkar, Andrew Szot, Erik Wijmans, Vladlen Koltun, Dhruv Batra, Kayvon Fatahalian
ACM Transactions on Graphics, 42(4), 2023

Habitat 2.0: Training Home Assistants to Rearrange their Habitat

Andrew Szot, Alexander Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John M Turner, Noah D Maestre, Mustafa Mukadam, Devendra Singh Chaplot, Oleksandr Maksymets, Aaron Gokaslan, Vladimír Vondruš, Franziska Meier, Wojciech Galuba, Angel X Chang, Zsolt Kira, Vladlen Koltun, Jitendra Malik, Manolis Savva, Dhruv Batra
Advances in Neural Information Processing Systems (NeurIPS), 2021
(Selected for spotlight presentation)

Learning to Drive from a World on Rails

Dian Chen, Vladlen Koltun, and Philipp Krähenbühl
International Conference on Computer Vision (ICCV), 2021 (Selected for full oral presentation)

Megaverse: Simulating Embodied Agents at One Million Experiences per Second

Aleksei Petrenko, Erik Wijmans, Brennan Shacklett, and Vladlen Koltun
International Conference on Machine Learning (ICML), 2021

Large Batch Simulation for Deep Reinforcement Learning

Brennan Shacklett, Erik Wijmans, Aleksei Petrenko, Manolis Savva, Dhruv Batra, Vladlen Koltun, Kayvon Fatahalian
International Conference on Learning Representations (ICLR), 2021

Rearrangement: A Challenge for Embodied AI

Dhruv Batra, Angel X. Chang, Sonia Chernova, Andrew J. Davison, Jia Deng, Vladlen Koltun, Sergey Levine, Jitendra Malik, Igor Mordatch, Roozbeh Mottaghi, Manolis Savva, Hao Su
Technical Report, arXiv:2011.01975, 2020

Sample Factory: Egocentric 3D Control from Pixels at 100000 FPS with Asynchronous Reinforcement Learning

Aleksei Petrenko, Zhehui Huang, Tushar Kumar, Gaurav Sukhatme, and Vladlen Koltun
International Conference on Machine Learning (ICML), 2020

Learning by Cheating

Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Krähenbühl
Conference on Robot Learning (CoRL), 2019

Habitat: A Platform for Embodied AI Research

Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, Dhruv Batra
International Conference on Computer Vision (ICCV), 2019 (Nominated for the Best Paper Award)

Benchmarking Classic and Learned Navigation in Complex 3D Environments

Dmytro Mishkin, Alexey Dosovitskiy, and Vladlen Koltun
Technical Report, arXiv:1901.10915, 2019

Does Computer Vision Matter for Action?

Brady Zhou, Philipp Krähenbühl, and Vladlen Koltun
Science Robotics, 4(30), 2019

Assessing Generalization in Deep Reinforcement Learning

Charles Packer, Katelyn Gao, Jernej Kos, Philipp Krähenbühl, Vladlen Koltun, Dawn Song
Technical Report, arXiv:1810.12282, 2019

On Evaluation of Embodied Navigation Agents

Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, Amir R. Zamir
Technical Report, arXiv:1807.06757, 2018

Motion Perception in Reinforcement Learning with Dynamic Objects

Artemij Amiranashvili, Alexey Dosovitskiy, Vladlen Koltun, and Thomas Brox
Conference on Robot Learning (CoRL), 2018

Driving Policy Transfer via Modularity and Abstraction

Matthias Müller, Alexey Dosovitskiy, Bernard Ghanem, and Vladlen Koltun
Conference on Robot Learning (CoRL), 2018

On Offline Evaluation of Vision-based Driving Models

Felipe Codevilla, Antonio López, Vladlen Koltun, and Alexey Dosovitskiy
European Conference on Computer Vision (ECCV), 2018

Semi-parametric Topological Memory for Navigation

Nikolay Savinov, Alexey Dosovitskiy, and Vladlen Koltun
International Conference on Learning Representations (ICLR), 2018

TD or not TD: Analyzing the Role of Temporal Differencing in Deep Reinforcement Learning

Artemij Amiranashvili, Alexey Dosovitskiy, Vladlen Koltun, and Thomas Brox
International Conference on Learning Representations (ICLR), 2018

End-to-end Driving via Conditional Imitation Learning

Felipe Codevilla, Matthias Müller, Antonio López, Vladlen Koltun, and Alexey Dosovitskiy
International Conference on Robotics and Automation (ICRA), 2018

MINOS: Multimodal Indoor Simulator for Navigation in Complex Environments

Manolis Savva, Angel X. Chang, Alexey Dosovitskiy, Thomas Funkhouser, and Vladlen Koltun
Technical Report, arXiv:1712.03931, 2017

CARLA: An Open Urban Driving Simulator

Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio López, and Vladlen Koltun
Conference on Robot Learning (CoRL), 2017

Learning to Act by Predicting the Future

Alexey Dosovitskiy and Vladlen Koltun
International Conference on Learning Representations (ICLR), 2017 (Selected for full oral presentation)