Evaluation of a static architectural conformance checking method in a line of computer games

We present an evaluation of a simple method to find architectural problems in a product line of computer games. The method uses dependencies (direct, indirect, or no) to automatically classify types in the implementation to high-level components in the product line architecture. We use a commercially available tool to analyse dependencies in the source code. The automatic classification of types is compared to a manual classification by the developer, and all mismatches are reported. To evaluate the method, we inspect the source code and look for a pre-defined set of architectural problems in all types. We compare the set of types that contained problems to the set of types where the manual and automatic classification disagreed to determine precision and recall. We also investigate what changes are needed to correct the found mismatches by either designing and implementing changes in the source code or refining the automatic classification. Our evaluation shows that the simple method is effective at detecting architectural problems in a product line of four games. The method is lightweight, customisable and easy to implement early in the development cycle.