Multi-Metric Validation Under Uncertainty for Multivariate Model Outputs and Limited Measurements

Abstract

Model validation for real-world systems involves multiple sources of uncertainty, multivariate model outputs, and often a limited number of measurement samples. These factors preclude the use of many existing validation metrics, or at least limit the ability of the practitioner to derive insights from computed metrics. This paper seeks to extend the area metric (univariate only) and the model reliability metric (univariate and multivariate) to account for these issues. The model reliability metric was found to be more extendable to multivariate outputs, whereas the area metric presented some difficulties. Metrics of different types (area and model reliability), dimensionality (univariate and multivariate), and objective (bias effects, shape effects, or both) are used together in a ‘multi-metric’ approach that provides a more informative validation assessment. The univariate metrics can be used for output-by-output model diagnosis and the multivariate metrics contributes an overall model assessment that includes correlation among the outputs. The extensions to the validation metrics in this paper address limited measurement sample size, improve the interpretability of the metric results by separating the effects of distribution bias and shape, and enhance the model reliability metric's tolerance parameter. The proposed validation approach is demonstrated with a bivariate numerical example and then applied to a gas turbine engine heat transfer model.