A user-centric quantum benchmarking test suite and evaluation framework

Abstract

This article proposes a benchmark testing set and evaluation system for quantum computers. Our tests do not focus on the topology of quantum computers or the specific implementation details of preparing quantum bits. Instead, we examine the overall performance of quantum computers from the perspective of users. Inspired by traditional computer benchmark tests such as SPECCPU2017, we integrate existing scalable quantum applications and algorithms to generate a testing set that covers algorithms such as search, machine learning, factorization, portfolio optimization, and entanglement state preparation, effectively simulating real workloads. By running the testing set, we can understand the performance of current quantum computers and generate a comprehensive score by combining our evaluation system, which consists of sub-scores of various backend features, including quantum gate error rate, entanglement between quantum bits, cross talk, and connectivity. These sub-scores are calculated based on the program features of the testing cases combined with their running results, where the program features are analyzed through the logical circuits of the testing cases. We incorporate Hellinger fidelity and polarization rescaling into each benchmark to calculate the fidelity of the running results. Through our evaluation system, researchers can be guided toward research directions and understand how far quantum computers are from solving practical problems.