A Multi-Lingual Benchmark for Property-Based Testing of Quantum Programs

Abstract

We present a multi-lingual benchmark for (property-based) testing of quantum programs. We report on the methodology used to design our benchmark and the rationale behind its design decisions. Our benchmark covers three major quantum programming languages, namely Qiskit, Cirq, and Q#. We curate our benchmark from languages documentations, open source repositories, and academic papers. In order to demonstrate the common logic of the algorithms included in our benchmark, we start from an implementation in one language (often Qiskit) and produce comparable implementations in the other two languages. We produce several properties and mutants for each program as a benchmark to measure the effectiveness of property-based testing frameworks. We reflect on the high-level quantum programming concepts offered in the three languages of our benchmark and their possible impact on testability and quality assurance.