Incomplete test vectors fail to detect obscure VoIP software errors

Abstract

Most ITU-T (i.e., international telecommunication union standardization sector) standards provide precise specifications for the proper operating behaviors of the systems they specify. However, such specifications are inappropriate for some standards such as the standards for audio coders used in VoIP. For such standards, ITU-T commonly supplies a set of input test data with corresponding correct output results. In this paper, we focus on the G.729 audio-coder algorithm. We use a version of G.729 code that can produce the bit-exact desired output for the given set of input test data to show that there can still be errors in the code even though the output matches the output in the ITU-T specification. We demonstrate that the given test vectors are not comprehensive enough to detect some of the obscure errors that can exist in the software. Therefore, we cannot rely solely on the given test vectors to test and validate our code.