A hybrid approach to 2D robotic map evaluation

Abstract

This article introduces the Temple Map Evaluation Toolkit (TMET), which is a tool for evaluating robotic maps produced by existing mapping algorithms. The toolkit performs ground truth based evaluation, i.e. it compares similarities between a map defined as ground truth and a target map. TMET allows for hybrid evaluation, since methods for pose based as well as grid based evaluation are implemented. For pose based evaluation, the user can define regions on the ground truth map which are handled as transformable sub-maps. TMET allows for evaluation of grid based maps as well as segment based maps, and therefore covers most of the representations of maps for existing mapping algorithms. The paper introduces the toolkit and the underlying design principles and algorithms. Experiments with maps from simulated as well as real world data are presented, demonstrating that the tool can be used to evaluate the quality of a map in a quantitative way.