How to compare algorithms for automated insulin delivery using different sensors?

Abstract

Recent caution has been advised against comparing automatic insulin delivery (AID) systems that use different sensors.^{1, 2} While the same two AID systems (Tandem Control IQ (CoIQ) and Minimed 780 G (MM780G)) have shown clear differences in time in range (TIR) outcomes, studies report no clinically relevant difference in HbA1c^{3, 4} or only minimal changes in HbA1c and TIR.^{5, 6} A simplified overview of these four studies is presented in Table 1. Of note, all TIR values are numerically higher in persons using MM780G (from 2.4% to 7.0% points), but all HbA1c values are minimally lower in persons using CoIQ (about 0.3% points). Tree studies^{3, 4, 6} provide information on sensor mean glucose or glucose monitoring indicator (GMI)⁷ over a 2-week period and HbA1c, which can be translated into estimated p-glucose from the ADAG equation (A1c-derived average glucose).⁸ Although this indirect approach contains several pitfalls, the result may indicate differences in sensor performances. Specifically, the Dexcom sensor connected to CoIQ appears to measure glucose levels higher than the Guardian sensor connected to the MM780G. This observation aligns with a recent small-scale study in which persons used both sensors simultaneously, reporting a slight positive bias for the Dexcom and a negative bias for the Guardian sensor when compared to self-monitored blood glucose.⁹

This discrepancy between sensor-driven and sensor-independent glycaemic metrics indicates the influence of varying study populations and sensor performance across brands. However, does this necessarily imply that the insulin delivery algorithm and continuous glucose monitoring (CGM) data hold no significance?

Consider a scenario in which an insulin pump using algorithm 1 with sensor A is compared with an insulin pump using algorithm 2 with sensor B. Suppose users of algorithm 1 achieve a TIR of 68% and a sensor-based mean glucose of 9.0 mmol/L, while a comparable group of users of algorithm 2 achieves a TIR of 75% and a mean glucose of 8.5 mmol/L, yet with a similar HbA1c. In such cases, TIR and sensor mean glucose metrics may be misleading from a clinical perspective, as sensor A likely measures glucose levels higher than sensor B. If algorithm 1 receives glucose input from sensor B, this is a new product with unknown results requiring new validation.

The goal of developing algorithms for insulin delivery is to enhance safety, minimize the risk of hypoglycaemia, ensure patient adherence and increase TIR, guided by input from the specific sensor. In this context, algorithm 2 is more effective than algorithm 1. A 7 percentage point difference in TIR could translate into a meaningful difference in the risk of diabetic complications.¹⁰ The apparent ‘rescue’ of algorithm 1 due to differences in sensor performance fails to capture the fundamental intellectual and clinical objectives underlying algorithm design.

Sensor performance has traditionally been reported as the mean absolute relative difference (MARD) between a sensor and a reference glucose concentration from a blood sample. There is a growing understanding that a similar and low MARD value does not guarantee the absence of any clinically relevant differences between sensors.^{2, 11} One of the drawbacks of MARD is that this metric does not provide information about the direction of any deviation from the reference value.¹²

To reliably evaluate the clinical performance of different AID systems based solely on CGM data, an internationally accepted standardization of glucose sensors is essential, with adherence mandated for manufacturers.^13-15 However, this standardization process is complicated by the lack of a definitive reference value for interstitial glucose and the inherent difficulties in estimating plasma glucose from interstitial glucose under varying physiological conditions.¹³

While we fully agree that sensor standardization is essential, and HbA1c is still needed for clinical assessment, we believe that insulin delivery algorithms per se can still be evaluated effectively with sensor-based data despite differences in sensor performance. In other words, the insulin delivery algorithm, as a data engineering product, can be evaluated in its own right from the sensor data that served as input during its development and optimisation process.

KWH has received grants for an investigator-initiated study from Abbott Diabetes Care and Novo Nordisk.