Visualizing the value of diagnostic tests and prediction models, part III. Numerical example with discrete risk groups and miscalibration.

Abstract

Background and objectives: In this third of a 3-part series, we use net benefit (NB) graphs to evaluate a risk model that divides D-dimer results into 8 intervals to estimate the probability of pulmonary embolism (PE). This demonstrates the effect of miscalibration on NB graphs.

Method: We evaluate the risk model's performance using pooled data on 6013 participants from 5 PE diagnostic management studies. For a range of values of the "exchange rate" (w, the treatment threshold odds), we obtained NB of applying the risk model by subtracting the number of unnecessary treatments weighted by the exchange rate from the number of appropriate treatments and then dividing by the population size.

Results: In NB graphs, in which the x-axis is scaled linearly with the exchange rate w, miscalibration causes vertical changes in NB. If the risk model overestimates risk, as in this example, the NB graph for the risk model has vertical jumps up. These are due to the sudden gain in NB resulting from less overtreatment when the treatment threshold first exceeds the overestimated predicted risks.

Conclusion: Calculating NB is a logical approach to quantifying the value of a diagnostic test or risk prediction model. In the same dataset at the same treatment threshold probability, the risk model with the higher net benefit is the better model in that dataset. Most net benefit calculations omit the harm of doing the test or applying the risk model, but if it is nontrivial, this harm can be subtracted from the net benefit.