Interpretation and report writing in forensic comparisons of paint evidence: An interlaboratory exercise

This interlaboratory study evaluated a guide for interpreting and reporting trace evidence examinations. The online survey aimed to assess the examiners' interpretation of casework scenarios designed by a subject matter expert panel (SMEP), specifically for paint evidence. A pool of 30 scenarios was created, and 15 were assigned to each participant using multi-factor design to evaluate agreement among examiners on case sets with different conclusion ranges and difficulty levels. Exploratory data analysis and three generalized mixed-effects models were used to assess the data. From the 1267 responses received from 85 participants, approximately 93% of responses were consistent between participants and within the SMEP consensus and the next best category, while 73% agreed with the SMEP consensus that was considered the ground truth. Most disagreements were observed in worst-case scenarios created with intended higher difficulty and complex circumstances.

The statistical models showed a strong positive relationship between the reported and expected conclusions, indicating that participants' findings align with the SMEP consensus. On the other hand, the exercise's difficulty level and participant's experience did not have a significant impact on the reported conclusions. However, the credible intervals for the probabilities of the different reported conclusions indicate that more experienced participants achieve greater consensus for a given exercise. The consensus reached among practitioners represents an advance in the trace community’s efforts to standardize reporting of results and opinions when following systematic criteria.