MLino bench: A comprehensive benchmarking tool for evaluating ML models on edge devices

In today’s rapidly evolving technological landscape, Machine Learning (ML) has become an integral part of our daily lives, ranging from recommendation systems to advanced medical diagnostics and autonomous vehicles. As ML continues to advance, its applications extend beyond conventional boundaries. With the continuous refinement of the models and frameworks, the possibilities for leveraging these technologies into devices that traditionally lacked any form of computational autonomy are ever-expanding. This shift towards embedding ML capabilities directly into edge devices brings new challenges due to the stringent limitations these devices have in terms of memory, power consumption, and cost. The ML models implemented on such devices must find an equilibrium between memory footprint and performance, attaining a classification time that fulfills real-time demands and maintains a similar level of accuracy as the desktop version. Without automated assistance in managing these considerations, the complexity of evaluating multiple models can lead to suboptimal decisions.

In this paper, we introduce MLino Bench, an open-source benchmarking tool tailored for assessing lightweight ML models on edge devices with limited resources and capabilities. The tool accommodates various models, frameworks, and platforms, presenting a meticulous design that enables a comprehensive evaluation directly on the target device. It encompasses crucial metrics such as on-target accuracy, classification time, and model size, providing a versatile framework that assists practitioners in decision-making when deploying models to such devices.

The tool employs a fully streamlined benchmark flow involving training the ML model in a high-level interpreted language, porting, compiling, flashing, and finally benchmarking on the actual target. Our experimental evaluation of the tool highlights its flexibility in assessing multiple ML models across different model hyperparameters, frameworks, datasets, and embedded platforms. Furthermore, a distinctive advantage compared to state-of-the-art ML benchmarking tools is the inclusion of classical ML models, including Random Forests, Decision Trees, Support Vector Machines, Naive Bayes, and more. This sets our tool apart from others that predominantly emphasize only neural network models. Due to this inclusive approach, our tool facilitates the evaluation of ML models across a broad spectrum of devices, ranging from resource-constrained edge devices to those with medium and advanced computational capabilities.