On the Failure of the Area Metric for Validation Exercises of Stochastic Simulations

Abstract

This paper discusses the application of the Area Metric to the quantification of modeling errors. The focus of the discussion is the effect of the shape of the two distributions on the result produced by the Area Metric. Two different examples that assume negligible experimental and numerical errors are presented: the first case has experimental and simulated quantities of interest defined by normal distributions that require the definition of a mean value and a standard deviation; the second example is taken from the V&V10.1 ASME Standard. The first example, shows that relatively small differences between the mean values are sufficient for the area metric to be insensitive to the standard deviation. Furthermore, the example of the V&V10.1 ASME Standard produces an Area Metric equal to the difference between the mean values of experiments and simulations. Therefore, the error quantification is reduced to a single number that is obtained from a simple difference of two mean values. This means that the Area Metric fails to reflect a dependence for the difference in the shape of the distributions representing variability. The paper also presents an alternative version of the Area Metric that does not filter the effect of the shape of the distributions by utilizing a reference simulation that has the same mean value of the experiments. This means that the quantification of the modeling error will have contributions from the difference in mean values and from the shape of the distributions.