Danny Driess

I am a Research Scientist building models that bring AI into the physical world.
Previously, I was a Senior Research Scientist at Google DeepMind, working in the intersection between Gemini and robotics.
I received my PhD from TU Berlin, my advisor was Marc Toussaint.
In 2020, I visited Russ Tedrake's Lab at MIT.
I was an intern at Google Brain hosted by Pete Florence.
I organized four RSS workshops in 2020, 2021, and 2023 discussing the synergies between learning and planning, and 2022 on Implicit Representations for Robotics.

Email: danny dot driess at gmail dot com

Google Scholar Twitter

Publications

For a full list of publications see Google Scholar

PaLM-E: An Embodied Multimodal Language Model

Danny Driess, Fei Xia, Mehdi Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, Pete Florence
ICML, 2023

PDF Videos arXiv

@inproceedings{driess2023palme,
    title={PaLM-E: An Embodied Multimodal Language Model},
    author={Driess, Danny and Xia, Fei and Sajjadi, Mehdi S. M. and Lynch, Corey and Chowdhery, Aakanksha and Ichter, Brian and Wahid, Ayzaan and Tompson, Jonathan and Vuong, Quan and Yu, Tianhe and Huang, Wenlong and Chebotar, Yevgen and Sermanet, Pierre and Duckworth, Daniel and Levine, Sergey and Vanhoucke, Vincent and Hausman, Karol and Toussaint, Marc and Greff, Klaus and Zeng, Andy and Mordatch, Igor and Florence, Pete},
    booktitle={Proceedings of the International Conference on Machine Learning},
    year={2023}
}
RT-2: Vision-Language-Action Models

Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexander Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Lisa Lee, Tsang-Wei Edward Lee, Sergey Levine, Yao Lu, Henryk Michalewski, Igor Mordatch, Karl Pertsch, Kanishka Rao, Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, Brianna Zitkovich
CoRL, 2023

arXiv Videos

@inproceedings{rt22023arxiv,
    title={RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control},
    author={Anthony Brohan and Noah Brown and Justice Carbajal and Yevgen Chebotar and Xi Chen and Krzysztof Choromanski and Tianli Ding and Danny Driess and Avinava Dubey and Chelsea Finn and Pete Florence and Chuyuan Fu and Montse Gonzalez Arenas and Keerthana Gopalakrishnan and Kehang Han and Karol Hausman and Alex Herzog and Jasmine Hsu and Brian Ichter and Alex Irpan and Nikhil Joshi and Ryan Julian and Dmitry Kalashnikov and Yuheng Kuang and Isabel Leal  and Lisa Lee and Tsang-Wei Edward Lee and Sergey Levine and Yao Lu and Henryk Michalewski and Igor Mordatch and Karl Pertsch and Kanishka Rao and Krista Reymann and Michael Ryoo and Grecia Salazar and Pannag Sanketi and Pierre Sermanet and Jaspiar Singh and Anikait Singh and Radu Soricut and Huong Tran and Vincent Vanhoucke and Quan Vuong and Ayzaan Wahid and Stefan Welker and Paul Wohlhart and  Jialin Wu and Fei Xia and Ted Xiao and Peng Xu and Sichun Xu and Tianhe Yu and Brianna Zitkovich},
    booktitle={Proc{.} of the Annual Conf{.} on Robot Learning (CoRL)},
    year={2023}
}
Reinforcement Learning with Neural Radiance Fields

Danny Driess, Ingmar Schubert, Pete Florence, Yunzhu Li, Marc Toussaint
NeurIPS, 2022

PDF Video arXiv

@InProceedings{22-driess-NeRF-RL,
  title={Reinforcement Learning with Neural Radiance Fields},
  author={Driess, Danny and Schubert, Ingmar and Florence, Pete and Li, Yunzhu and Toussaint, Marc},
  booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
  year={2022}
}
Learning Multi-Object Dynamics with Compositional Neural Radiance Fields

Danny Driess, Zhiao Huang, Yunzhu Li, Russ Tedrake, Marc Toussaint
CoRL, 2022

PDF Video arXiv

@InProceedings{2022-driess-compNerf,
  title = {Learning Multi-Object Dynamics with Compositional Neural Radiance Fields},
  author = {Driess, Danny and Huang, Zhiao and Li, Yunzhu and Tedrake, Russ and Toussaint, Marc},
  booktitle = {Proc{.} of the Annual Conf{.} on Robot Learning (CoRL)},
  pages = {1755--1768},
  year = {2023},
  volume = {205},
  series = {Proceedings of Machine Learning Research},
  month = {14--18 Dec},
  publisher = {PMLR},
}
Learning Models as Functionals of Signed-Distance Fields for Manipulation Planning

Danny Driess, Jung-Su Ha, Marc Toussaint, Russ Tedrake
CoRL 2021

PDF Video

@InProceedings{21-driess-CORL,
  title = {Learning Models as Functionals of Signed-Distance Fields for Manipulation Planning},
  author = {Driess, Danny and Ha, Jung-Su and Toussaint, Marc and Tedrake, Russ},
  booktitle = {Proc{.} of the Annual Conf{.} on Robot Learning (CoRL)},
  year = {2021},
}
Learning to Solve Sequential Physical Reasoning Problems from a Scene Image

Danny Driess, Jung-Su Ha, Marc Toussaint
IJRR 2021

PDF Video DOI

@article{21-driess-IJRR,
  title = {Learning to Solve Sequential Physical Reasoning Problems from a Scene Image},
  author = {Driess, Danny and Ha, Jung-Su and Toussaint, Marc},
  journal = {The International Journal of Robotics Research (IJRR)},
  year = {2021},
  doi = {10.1177/02783649211056967},
}
Learning Geometric Reasoning and Control for Long-Horizon Tasks from Visual Input

Danny Driess, Jung-Su Ha, Russ Tedrake, Marc Toussaint
ICRA 2021

PDF Video

@InProceedings{21-driess-ICRA,
  title = {Learning Geometric Reasoning and Control for Long-Horizon Tasks from Visual Input},
  author = {Danny Driess and Jung-Su Ha and Russ Tedrake and Marc Toussaint},
  booktitle = {Proc{.} of the IEEE Int{.} Conf{.} on Robotics and Automation (ICRA)},
  year = {2021}
}
Deep Visual Reasoning: Learning to Predict Action Sequences for Task and Motion Planning from an Initial Scene Image

Danny Driess, Jung-Su Ha, Marc Toussaint
R:SS 2020

PDF Video

@InProceedings{20-driess-RSS,
  title = {Deep Visual Reasoning: Learning to Predict Action Sequences for Task and Motion Planning from an Initial Scene Image},
  author = {Danny Driess and Jung-Su Ha and Marc Toussaint},
  booktitle = {Proc{.} of Robotics: Science and Systems (R:SS)},
  year = {2020}
}
Deep Visual Heuristics: Learning Feasibility of Mixed-Integer Programs for Manipulation Planning

Danny Driess, Ozgur Oguz, Jung-Su Ha, Marc Toussaint
ICRA 2020

Nominated for Best Paper Award in Automation

PDF Video

@InProceedings{20-driess-ICRA,
  title = {Deep Visual Heuristics: Learning Feasibility of Mixed-Integer Programs for Manipulation Planning},
  author = {Danny Driess and Ozgur Oguz and Jung-Su Ha and Marc Toussaint},
  booktitle = {Proc{.} of the IEEE Int{.} Conf{.} on Robotics and Automation (ICRA)},
  year = {2020}
}
Probabilistic framework for constrained manipulations and task and motion planning under uncertainty

Jung-Su Ha, Danny Driess, Marc Toussaint
ICRA 2020

PDF Video

@InProceedings{20-ha-ICRA,
  title={Probabilistic framework for constrained manipulations and task and motion planning under uncertainty},
  author={Ha, Jung-Su and Driess, Danny and Toussaint, Marc},
  booktitle = {Proc{.} of the IEEE Int{.} Conf{.} on Robotics and Automation (ICRA)},
  year = {2020}
}
Optimality Principles in Human Point-to-Manifold Reaching Accounting for Muscle Dynamics

Isabell Wochner, Danny Driess, Heiko Zimmermann, Daniel F. B. Haeufle, Marc Toussaint, Syn Schmitt
Frontiers in Computational Neuroscience 2020

Paper

@ARTICLE{20-wochner-frontiers,
  AUTHOR={Wochner, Isabell and Driess, Danny and Zimmermann, Heiko and Haeufle, Daniel F. B. and Toussaint, Marc and Schmitt, Syn},
  TITLE={Optimality Principles in Human Point-to-Manifold Reaching Accounting for Muscle Dynamics},
  JOURNAL={Frontiers in Computational Neuroscience},
  VOLUME={14},
  PAGES={38},
  YEAR={2020},
}
Hierarchical Task and Motion Planning using Logic-Geometric Programming (HLGP)

Danny Driess, Ozgur Oguz, Marc Toussaint
R:SS 2020 Workshop on Robust Task and Motion Planning

PDF Video

@Misc{19-driess-RSSws,
  title = {Hierarchical Task and Motion Planning using Logic-Geometric Programming (HLGP)},
  author = {Danny Driess and Ozgur Oguz and Marc Toussaint},
  year = {2019},
  howpublished = {RSS Workshop on Robust Task and Motion Planning}
}
Active Inverse Model Learning with Error and Reachable Set Estimates

Danny Driess, Syn Schmitt, Marc Toussaint
IROS 2019

PDF Video

@InProceedings{19-driess-IROS,
  title = {Active Inverse Model Learning with Error and Reachable Set Estimates},
  author = {Danny Driess and Syn Schmitt and Marc Toussaint},
  booktitle = {Proc{.} of the IEEE Int{.} Conf{.} on Intelligent Robots and Systems (IROS)},
  year = {2019}
}
Active Multi-Contact Continuous Tactile Exploration with Gaussian Process Differential Entropy

Danny Driess, Daniel Hennes, Marc Toussaint
ICRA 2019

PDF Video

@InProceedings{19-driess-ICRA,
  title = {Active Multi-Contact Continuous Tactile Exploration with Gaussian Process Differential Entropy},
  author  = {Danny Driess and Daniel Hennes and Marc Toussaint},
  booktitle  = {Proc{.} of the IEEE Int{.} Conf{.} on Robotics and Automation (ICRA)},
  year = {2019},
}
Learning to Control Redundant Musculoskeletal Systems with Neural Networks and SQP: Exploiting Muscle Properties

Danny Driess, Heiko Zimmermann, Simon Wolfen, Dan Suissa, Daniel Haeufle, Daniel Hennes, Marc Toussaint, Syn Schmitt
ICRA 2018

PDF Video

@InProceedings{18-driess-ICRA,
  title = {Learning to Control Redundant Musculoskeletal Systems with Neural Networks and {SQP}: Exploiting Muscle Properties},
  author = {Danny Driess and Heiko Zimmermann and Simon Wolfen and Dan Suissa and Daniel Haeufle and Daniel Hennes and Marc Toussaint and Syn Schmitt},
  booktitle = {Proc{.} of the IEEE Int{.} Conf{.} on Robotics and Automation (ICRA)},
  year = {2018}
}
Active Learning with Query Paths for Tactile Object Shape Exploration

Danny Driess, Peter Englert, Marc Toussaint
IROS 2017

PDF Video

@InProceedings{17-driess-IROS,
  author = {Danny Driess and Peter Englert and Marc Toussaint},
  title = {Active Learning with Query Paths for Tactile Object Shape Exploration},
  booktitle = {Proc. of the IEEE Int. Conf on Intelligent Robots and Systems (IROS)},
  year = {2017}
}
Constrained Bayesian Optimization of Combined Interaction Force/Task Space Controllers for Manipulations

Danny Driess, Peter Englert, Marc Toussaint
ICRA 2017

PDF Video

@InProceedings{17-driess-ICRA,
  author  = {Danny Driess and Peter Englert and Marc Toussaint},
  title = {Constrained Bayesian Optimization of Combined Interaction Force/Task Space Controllers for Manipulations},
  booktitle  = {Proc. of the IEEE Int. Conf. on Robotics and Automation (ICRA)},
  year = {2017}
}