Empirical evaluation of reparability scoring systems for validity and reliability

Abstract

The validity and reliability of four prevalent reparability scoring systems has been investigated by comparing scores of ten smart phones and six vacuum cleaners versus empirically measured repair times, as well as comparing hypothetical ideal and problematic scenarios. Ease of disassembly methods was also assessed for five smart TVs, four washing machines and six vacuum cleaners. The scoring systems studied were the French Reparability Index (FRI), Joint Research Centre Scoring System (RSS/JRC), iFixit, and ONR19202. Overall scores of products across scoring systems were relatively well correlated, indicating a fair amount of overall reliability. However, the variability in scores for the best and worst case of the same product was often larger than the differences between products. Validity was good for products that are easily repairable, but scorecards often failed to score low when repair is infeasible or too expensive. Repair scores greatly depend on disassembly; since some scorecards count numbers of disassembly steps and other scorecards use proxy times, these two methods were compared against empirical disassembly times for five vacuum cleaners, five televisions, and four washing machines. The proxy time method was found to be highly accurate for all three product categories; the steps method was less so. It indicated the relative ease of disassembly well for washing machines, but not for televisions or vacuum cleaners. Finally, this study proposes improvements to scoring methods, including a limiting factor approach and the development of clearer protocols, to ensure the scoring systems are robust, reliable, and can effectively guide sustainable product design.