Evaluating Motion Planning Performance

Speakers

• 

  Prof. Dmitry Berenson

  Learning Where to Trust Unreliable Dynamics Models for Motion Planning

  Abstract: This talk will present our recent work on using unreliable dynamics models for motion planning. Given a dynamics model, our method determines where in state-action space that model can be trusted and then plans trajectories with statistical guarantees of validity. I will end with some thoughts on how such methods can be evaluated.

  Bio: Dmitry Berenson is an Associate Professor in the Electrical Engineering and Computer Science Department of the University of Michigan. His research focuses on algorithms that allow robots to interact with the world through general-purpose learning, motion planning, and manipulation. He is interested in the entire pipeline of algorithm development, from creating novel algorithms to proving their theoretical properties, evaluating them on physical systems, and distributing them to open-source communities.

• 

  Prof. Xuesu Xiao

  Evaluating Motion Planning “in-the-Loops”

  Abstract: When evaluating motion planning performance, most researchers treat motion planning as a standalone problem, i.e., given accurate world representations as input and reliable controllers to execute planning output, how does the planner perform based on planning specific metrics, such as optimality, convergence, clearance, time, and complexity. However, such evaluation may not directly reflect how the planner would perform when being used in a robot system or in an entire robot fleet. In this talk, we will situate motion planners in two nested loops: the inner development (or sense-plan-act) loop is usually plagued with inaccurate world representations, imperfect actuation, uncertain real-world conditions, and sometimes unmodeled human interactions; the outer deployment loop is where most motion planners interface with real-world tasks, mostly in industrial settings, and go through a sequence of testing, integration, deployment, and improvement/rollback. We will discuss why it is necessary to situate motion planning evaluation in both loops, not only from an academic perspective, but also an industrial viewpoint. We will also discuss our vision in bridging the gap between robotics researchers and practitioners when evaluating motion planning performance with the aim of jointly creating highly capable motion planners that can work in the real world at scale.

  Bio: Xuesu Xiao is an Assistant Professor in the Department of Computer Science at George Mason University. Xuesu (Prof. XX) directs the RobotiXX lab, in which researchers (XX-Men) and robots (XX-Bots) work together at the intersection of motion planning and machine learning with a specific focus on robustly deployable field robotics. Xuesu is also a Roboticist at Everyday Robots, a company born from X, the moonshot factory, working alongside teams at Google and building robots that can learn by itself to help anyone with (almost) anything.

  Video

• 

  Prof. Anca Dragan

  Intro to Experiment Design for Motion Planning

  Abstract: I'll go over the basics of what experiment design is, what makes a good experiment and what to watch out for, as well as my own opinions for how to evaluate motion planning performance. I'll draw on examples from my own early grad school days, when I was working on motion planning, but learning about user studies and what principles can be ported over to evaluate algorithmic contributions in robotics. I hope it will be helpful!

  Bio: Anca Dragan is an Associate Professor in the EECS Department at UC Berkeley. Her goal is to enable robots to work with, around, and in support of people. She runs the InterACT Lab, where she focuses on algorithms for human-robot interaction. She also helped found and serve on the steering committee for the Berkeley AI Research (BAIR) Lab, and is a co-PI of the Center for Human-Compatible AI. Anca has been honored by the Sloan Fellowship, MIT TR35, the Okawa award, an NSF CAREER award, and the PECASE award.

  Video

• 

  Dr. Adithya Murali

  Lessons for Benchmarking from Learning Motion Planners

  Abstract: The ingredients needed for benchmarking motion planning and those required to effectively train motion planners are very similar. In this talk, we focus on several insights we gained from our efforts to learn motion planners via large-scale simulation, which can be used to help build community-wide benchmarks.

  Bio: Adithya Murali is a research scientist ad the NVIDIA Seattle Robotics Lab. He received his PhD at The Robotics Institute, Carnegie Mellon University advised by Abhinav Gupta, where he was supported by the Uber Presidential Fellowship. During his PhD, he also worked at NVIDIA and spent time at Facebook, Amazon, LBL and NTU. His general interests are in robotic manipulation, perception and learning and also in building robot systems. (edited)

  Video

• 

  Dr. Clemens Eppner

  Lessons for Benchmarking from Learning Motion Planners

  Abstract: The ingredients needed for benchmarking motion planning and those required to effectively train motion planners are very similar. In this talk, we focus on several insights we gained from our efforts to learn motion planners via large-scale simulation, which can be used to help build community-wide benchmarks.

  Bio: Clemens Eppner is a Research Scientist in the Seattle Robotics Lab at NVIDIA Research. He is particularly interested in problems surrounding robotic grasping and manipulation, including aspects of planning, control, and perception. Before joining NVIDIA, he received his Ph.D. at the Robotics and Biology Lab at TU Berlin.

  Video

• 

  Dr. Mark Moll

  Hyperparameter Optimization as a Tool for Motion Planning Algorithm Selection

  Abstract: Over the years, many motion planning algorithms have been proposed. It is often unclear which algorithm might be best suited for a particular class of problems. The problem is compounded by the fact that algorithm performance can be highly dependent on parameter settings. This paper shows that hyperparameter optimization is an effective tool in both algorithm selection and parameter tuning over a given set of motion planning problems. We present different loss functions for optimization that capture different notions of optimality. The approach is evaluated on a broad range of scenes using two different manipulators, a Fetch and a Baxter. We show that optimized planning algorithm performance significantly improves upon baseline performance and generalizes broadly in the sense that performance improvements carry over to problems that are very different from the ones considered during optimization.

  Bio: Mark Moll is the Director of Research at PickNik, a robotics software development and consultancy company that is supporting the MoveIt motion planning framework. He has worked in robotics for more than 20 years, with a focus on motion planning. For most of that time he was a senior research scientist in the Computer Science Department at Rice University, where he lead the development of the Open Motion Planning Library (OMPL), which is widely used in industry and academic research (often via MoveIt / ROS). He has over 80 peer-reviewed publications with research contributions in applied algorithms for problems in robotics and computational structural biology. He has extensive experience deploying novel algorithms on a variety of robotic platforms, ranging from NASA’s Robonaut 2 to autonomous underwater vehicles and self-reconfigurable robots.

  Video

• 

  Dr. Jonathan Gammell

  Nonparametric Statistical Evaluation of Sampling-Based Motion Planning Algorithms

  Abstract: Sampling-based algorithms are important approaches in motion planning. Their use of sampling sequences allows them to quickly find good solutions to many problems but makes their individual performance nondeterministic. Algorithms are instead evaluated by running independent trials and performing appropriate statistical analysis on the distribution of results. This talk will share best practices for statistical evaluation of anytime and nonanytime sampling-based planning algorithms. It will present performance metrics and associated nonparametric confidence intervals to quantify planner performance without making any assumptions about the underlying distribution of results. These metrics allow algorithms to be evaluated in a fair and meaningful way and implementations are available in the open-source Planner Developer Tools (PDT).

  Bio: Jonathan Gammell is a Departmental Lecturer in Robotics at the Oxford Robotics Institute (ORI). He leads the Estimation, Search, and Planning (ESP) research group which seeks to develop and exploit better understandings of fundamental robotic problems. He holds a Ph.D. and M.A.Sc. in Aerospace Science & Engineering from the University of Toronto (UTIAS) and a B.A.Sc. in Mechanical Engineering (Co-op) with a Physics Option from the University of Waterloo. Jonathan is a dedicated 'full-stack' roboticist with extensive experience solving real-world problems with robotic hardware and software. He has deployed autonomous systems around the world on a variety of projects.

  Video

• 

  Prof. Hanna Kurniawati

  Comparable Performance Evaluation in Motion Planning under Uncertainty

  Abstract: Uncertainty is ubiquitous and increases the challenge of creating reliable and comparable performance evaluation in motion planning. In this talk, I will review some of these challenges. I will also discuss our effort in developing a software tool to alleviate the above difficulties for motion planning under uncertainty, specifically when uncertainty is caused by non-deterministic effects of actions and errors in sensors and sensing.

  Bio: Hanna Kurniawati is an Associate Professor with ANU and CS Futures Fellowship at the Research School of Computer Science, Australian National University (ANU), as well as a CI and exec of a major interdisciplinary project Humansing Machine Intelligence. Her research focuses on algorithms to enable robust decision theory to become practical software tools, with applications in robotics and the assurance of autonomous systems. Such software tools will enable robots to design their own strategies, such as deciding what data to use, how to gather the data, and how to move, for accomplishing various tasks well, despite various modelling errors and types of uncertainty, and despite limited to no information about the system and its operating environment.

  Video

• 

  Dr. Andreas Orthey

  Benchmarking for Motion Planning Applications

  Abstract: Benchmarking is often thought to be a crucial component to evaluate the goodness of a given motion planning algorithm. However, most motion planning applications require different benchmarking frameworks and tools to properly evaluate them. I will showcase this on three different motion planning applications in spot welding, bin picking, and rearrangement planning. Because those applications have different requirements, we can observe not only different planning behavior, but we can often significantly improve upon existing planners by designing tailor-made solutions for specific benchmark data sets. This has not only ramifications for how we should design a benchmark dataset, but also for the development of novel planning algorithms.

  Bio: Andreas Orthey is a staff robotics scientist at Realtime Robotics. He was previously a postdoctoral researcher with Marc Toussaint at the Max Planck Institute for Intelligent Systems (MPI-IS).

  Video