Evaluating Motion Planning Performance

Metrics, Tools, Datasets, and Experimental Design

October 23, 2022

IEEE/RSJ IROS 2022

Kyoto, Japan


Evaluating Motion Planning Performance

Motion planning research has produced a plethora of techniques with distinct strengths and weaknesses, and widespread applications in different areas of robotics, such as autonomous driving, mobile manipulation, and locomotion. However, the field largely lacks standardized datasets and performance metrics for comparison. As a result, researchers resort to developing their own ad-hoc experimental designs, which can be time-consuming, prone to bias, and narrow in scope. This matter makes direct comparison of approaches against the state-of-the-art difficult. Additionally, the integration of machine learning methods with motion planners has further increased the demand for large common training datasets with a rich distribution of problems. This workshop is concerned with challenges in providing reliable, consistent, and comparable performance evaluation of motion planning.

This workshop will bring together robotics researchers and practitioners in academia and industry interested in motion planning. The workshop program will include invited talks, panel discussions, and presentation of contributed papers through lightning talks and a poster session. The panels will examine two core topics: reproducible experimental design and informative evaluation metrics. The experimental design panel will center around dataset creation and tools for benchmarking, and will assess the current state of motion planner experimentation. The evaluation metrics panel will discuss the different performance metrics and criteria that are currently in use to evaluate planners and consider other potential metrics. In order to facilitate the above goals, we will collect and showcase a set of tools, documentation, and datasets provided by the community that will be made available on the workshop website. The documentation will include introductory guides on integrating new motion planners with existing infrastructure, creating new motion planning datasets, and using new or existing datasets for benchmarking planners.

Schedule

The workshop schedule may change as we draw closer to the event.

Time (GMT+9) Session Speaker(s)
09:00 - 09:30 Opening remarks Organizers
Session 1: Reproducible Experimental Design
09:30 - 09:50 Invited talk 1 Xuesu Xiao George Mason University
09:50 - 10:10 Invited talk 2 Andreas Orthey Realtime Robotics
10:10 - 10:30 Invited talk 3 Anca Dragan UC Berkeley
10:30 - 10:50 Invited talk 4 Dieter Fox NVIDIA Robotics, University of Washington
10:50 - 11:05 Coffee break
11:05 - 11:50 Panel on reproducible experimental design Speakers 1-4
11:50 - 12:30 Lightning talks Paper presenters
12:30 - 13:30 Lunch
13:30 - 14:30 Poster session/Coffee Paper presenters
Session 2: Performance and Evaluation Metrics
14:30 - 14:50 Invited talk 5 Mark Moll PickNik Robotics
14:50 - 15:10 Invited talk 6 Jonathan Gammell Oxford Robotics Institute
15:10 - 15:30 Invited talk 7 Hanna Kurniawati Australian National University
15:30 - 15:50 Invited talk 8 Dmitry Berenson University of Michigan
15:50 - 16:05 Coffee break
16:05 - 16:50 Panel on performance and evaluation metrics Speakers 5-8
16:50 - 17:00 Closing remarks Organizers

Speakers

  • Prof. Xuesu Xiao

    Invited talk 1

    Prof. Xuesu Xiao

    Xuesu Xiao is an assistant Professor in the Department of Computer Science at GMU (Fall 2022). He has previously been a roboticist on the Everyday Robot Project at X, The Moonshot Factory, and a research affiliate of the Department of Computer Science at UT Austin

  • Dr. Andreas Orthey

    Invited talk 2

    Dr. Andreas Orthey

    Andreas Orthey is a staff robotics scientist at Realtime Robotics. He was previously a postdoctoral researcher with Marc Toussaint at the Max Planck Institute for Intelligent Systems (MPI-IS).

  • Prof. Anca Dragan

    Invited talk 3

    Prof. Anca Dragan

    Anca Dragan is an Associate Professor in the EECS Department at UC Berkeley. Her goal is to enable robots to work with, around, and in support of people. She runs the InterACT Lab, where she focuses on algorithms for human-robot interaction. She also helped found and serve on the steering committee for the Berkeley AI Research (BAIR) Lab, and is a co-PI of the Center for Human-Compatible AI. Anca has been honored by the Sloan Fellowship, MIT TR35, the Okawa award, an NSF CAREER award, and the PECASE award.

  • Prof. Dieter Fox

    Invited talk 4

    Prof. Dieter Fox

    Dieter Fox is a Professor in the Allen School of Computer Science and Engineering at the University of Washington. He additionally leads NVIDIA's Robotics Research team, and is a Fellow of the AAAI, ACM, and IEEE, as well as recipient of the IEEE RAS Pioneer Award. His research focuses on perception and its connection to control, with the goal of enabling systems to interact with people and their environment in an intelligent way.

  • Dr. Mark Moll

    Invited talk 5

    Dr. Mark Moll

    Mark Moll is the Director of Research at PickNik, a robotics software development and consultancy company that is supporting the MoveIt motion planning framework. He has worked in robotics for more than 20 years, with a focus on motion planning. For most of that time he was a senior research scientist in the Computer Science Department at Rice University, where he lead the development of the Open Motion Planning Library (OMPL), which is widely used in industry and academic research (often via MoveIt / ROS). He has over 80 peer-reviewed publications with research contributions in applied algorithms for problems in robotics and computational structural biology. He has extensive experience deploying novel algorithms on a variety of robotic platforms, ranging from NASA’s Robonaut 2 to autonomous underwater vehicles and self-reconfigurable robots.

  • Dr. Jonathan Gammell

    Invited talk 6

    Dr. Jonathan Gammell

    Jonathan Gammell is a Departmental Lecturer in Robotics at the Oxford Robotics Institute (ORI). He leads the Estimation, Search, and Planning (ESP) research group which seeks to develop and exploit better understandings of fundamental robotic problems. He holds a Ph.D. and M.A.Sc. in Aerospace Science & Engineering from the University of Toronto (UTIAS) and a B.A.Sc. in Mechanical Engineering (Co-op) with a Physics Option from the University of Waterloo. Jonathan is a dedicated 'full-stack' roboticist with extensive experience solving real-world problems with robotic hardware and software. He has deployed autonomous systems around the world on a variety of projects.

  • Prof. Hanna Kurniawati

    Invited talk 7

    Prof. Hanna Kurniawati

    Hanna Kurniawati is an Associate Professor with ANU and CS Futures Fellowship at the Research School of Computer Science, Australian National University (ANU), as well as a CI and exec of a major interdisciplinary project Humansing Machine Intelligence. Her research focuses on algorithms to enable robust decision theory to become practical software tools, with applications in robotics and the assurance of autonomous systems. Such software tools will enable robots to design their own strategies, such as deciding what data to use, how to gather the data, and how to move, for accomplishing various tasks well, despite various modelling errors and types of uncertainty, and despite limited to no information about the system and its operating environment.

  • Prof. Dmitry Berenson

    Invited talk 8

    Prof. Dmitry Berenson

    Dmitry Berenson is an Associate Professor in the Electrical Engineering and Computer Science Department of the University of Michigan. His research focuses on algorithms that allow robots to interact with the world through general-purpose learning, motion planning, and manipulation. He is interested in the entire pipeline of algorithm development, from creating novel algorithms to proving their theoretical properties, evaluating them on physical systems, and distributing them to open-source communities.

Submission

We invite submissions of extended abstracts describing contributions in topics broadly related to evaluation and benchmarking of motion planning across all areas of robotics (including manipulation, mobile robots, navigation, field robotics, etc.). Such contributions may convey, e.g., novel motion planner evaluation and benchmarking methodologies, case-study evaluations of new and existing planners, reproducibility studies, and short opinion papers. Submission of reports on new datasets and open-source tools is also greatly encouraged. Dataset and tool submissions should include a link to a well-documented, open-source repository.

These submissions will help us a) expand the set of evaluation tools available to motion planning researchers, b) exhibit methods for benchmarking new planners against the state-of-the-art, and c) highlight the limitations of current benchmarking and evaluation methods. The contributed papers will be encouraged to use the collected open-source tools or make their experimental design compatible.

Authors of accepted submissions will be invited to present their work as a poster and in a “lightning talk” session during the workshop. Accepted abstracts will be posted on the workshop website; relevant tools and resources will be featured in a resource list on the workshop website and advertised to the community. The organizers intend to facilitate virtual participation and presentation as needed in close coordination with the conference organizers.

Extended abstracts may describe work currently under review (or recently appearing) in other robotics publication venues, in compliance with the author restrictions specified by such venues. Submissions will be judged on technical quality, relevance, significance, and clarity. Submitted abstracts should be 1–4 pages in length (including references and figures), following the IROS 2022 formatting guidelines. For reference:

Submission portal link

Please submit contributions through CMT: https://cmt3.research.microsoft.com/EMPP2022.

Important Dates

All deadlines are Pacific Daylight Time.

Submission Deadline: 23:59 PDT, September 23, 2022

Review Decisions: September 30, 2022

Deadline for Final Revisions: 23:59 PDT October 7, 2022

Workshop Date: October 23, 2022

Resources

OMPL

The Open Motion Planning Library(OMPL), consists of many state-of-the-art sampling-based motion planning algorithms. OMPL itself does not contain any code related to, e.g., collision checking or visualization. This is a deliberate design choice, so that OMPL is not tied to a particular collision checker or visualization front end. The library is designed so it can be easily integrated into systems that provide the additional needed components.

Paper: https://ieeexplore.ieee.org/abstract/document/6377468

Repository: http://ompl.kavrakilab.org/

Robowflex

Robowflex is a wrapper around MoveIt , DART , and other popular robotics libraries that offers an easy API for setting up and running motion planning experiments in isolation from a ROS system, while still using ROS components where needed.

Paper: https://arxiv.org/abs/2112.06402

Repository: https://github.com/KavrakiLab/robowflex

MotionBenchMaker

MotionBenchMaker is a ROS package that provides a comprehensive set of pre-generated datasets of motion planning problems and tools for synthesizing additional such problems and datasets.

Paper: https://arxiv.org/abs/2103.12826

Repository: https://github.com/KavrakiLab/motion_bench_maker

PlannerArena and OMPLBenchmarking

PlannerArena can be used to easily visualize benchmarking results obtained using the OMPL benchmarking format. Plannerarena can visualize both overall performance plots and progress plots, for the different planners. Both Robowflex and MotionBenchMaker are compatible with PlannerArena.

Paper: http://www.kavrakilab.org/publications/moll-sucan2015benchmarking-motion-planning.pdf

Website: http://plannerarena.org/

BARN

The BARN dataset "provides a suite of simulation environments to test collision-free mobile robot navigation in highly-cluttered environments. BARN focuses on testing a mobile robot's low-level motion skills (i.e. how to navigate), instead of task-level decision-making (i.e. where to navigate). These environments cover a wide variety of metric navigation difficulties, ranging from relative open spaces to extremely cluttered environments, where robots need to squeeze through dense obstacles without collisions. These difficulties represent challenging and adversarial environments for autonomous navigation in the real-world, e.g., post-disaster scenarios, such as search and rescue missions, and cause problems for the state-of-the-art navigation systems."

Paper: https://www.cs.utexas.edu/~xiao/papers/navdiffdataset_ssrr.pdf

Website: https://www.cs.utexas.edu/~xiao/BARN/BARN.html

Bench-MR

Bench-MR is a software suite of components that allow for the benchmarking of motion planning algorithms for wheeled mobile robots on various types of scenarios. The planners can use a large variety of extend functions, post-smoothing methods, and optimization objectives. Through a front-end based on Jupyter notebooks, Bench-MR provides tools for plotting and evaluation to gain insights into many aspects of the planning pipeline.

Paper: https://ieeexplore.ieee.org/document/9387068

Website: https://github.com/robot-motion/bench-mr

Endorsements

We have received endorsements from the following technical committees

IEEE RAS TC on Software Engineering for Robotics and Automation

IEEE RAS TC on Mobile Manipulation

IEEE RAS TC on Verification of Autonomous Systems

IEEE RAS TC on Algorithms for Planning and Control of Robot Motion

IEEE RAS TC on Performance Evaluation & Benchmarking of Robotic and Automation Systems

Organizers

Contact

Please contact the organizers at motionplanningworkshop@gmail.com with any questions.