Testing-what's missing? An incomplete list of challenges

Abstract

Summary form only given. As the testing area becomes mature, the challenges it poses shift. We describe some of these challenges and how they are addressed in recent works in various areas of testing. In recent years, the formulations of testing problems have changed from "given a problem, find a solution" to "given a problem and quality measures, find a high-quality solution". Quality guarantees in the form of lower and upper bounds and optimal solutions are derived, in addition to the more conventional demonstration of performance on benchmark circuits. Quality guarantees allow one to measure the distance between a given solution and an optimal solution, and provide criteria for evaluating a new procedure that are more effective than comparison to previously proposed procedures. We review several areas where bounds and optimal solutions have been found. Most procedures are specific to a given problem, and cannot be reused to solve other problems. In contrast, general-purpose paradigms allow a large variety of problems to be solved cost-effectively by plugging in the appropriate procedures into the same algorithm. Such paradigms allow faster program development and reuse of expertise acquired in solving other problems under the same paradigm. We describe several attempts at using existing paradigms and developing new ones, that successfully compete with special-purpose procedures. Recent works address testing issues at increasingly higher levels of the design cycle and offer an integrated treatment of design and test. High-level failure models are considered as well as solutions that are completely independent of a failure model. We describe some of these works and the advantages of the two directions. We conclude with an (incomplete) list of challenges for future research.