Quantitative probing: Validating causal models with quantitative domain knowledge

Abstract We propose quantitative probing as a model-agnostic framework for validating causal models in the presence of quantitative domain knowledge. The method is constructed in analogy to the train/test split in correlation-based machine learning. It is consistent with the logic of scientific discovery and enhances current causal validation strategies. The effectiveness of the method is illustrated using Pearl’s sprinkler example, before a thorough simulation-based investigation is conducted. Limits of the technique are identified by studying exemplary failing scenarios, which are furthermore used to propose a list of topics for future research and improvements of the presented version of quantitative probing. A guide for practitioners is included to facilitate the incorporation of quantitative probing in causal modelling applications. The code for integrating quantitative probing into causal analysis, as well as the code for the presented simulation-based studies of the effectiveness of quantitative probing are provided in two separate open-source Python packages.