Product effect size estimation and performance accuracy (validity and reliability) assessment for CATA and TCATA data

Abstract

Check-all-that-apply (CATA) questions are used frequently now in sensory and consumer research. Temporal check-all-that-apply (TCATA) is an extension of the CATA method into temporal sensory evaluation. How to estimate product effect size and how to assess performance accuracy (validity and reliability) are important topics for analysis of panel or consumer CATA and TCATA data. This paper originally uses Thurstonian discriminal distance d-prime, and two area measures: R-index and Gini-index to estimate product effect size and uses the intraclass correlation coefficient (ICC) and Cronbach's coefficient alpha (CA) to assess performance accuracy (validity and reliability). These two different but related topics are explored and combined in the paper. Based on estimated d-prime and CA values, the performance quality of CATA and TCATA data can be interpreted and named with colloquial labels as below: better performance accuracy (e.g., CA > 0.7) versus smaller product effect size (e.g., d-prime <0.74) suggests the performance is “Exciting”; better performance accuracy (e.g., CA > 0.7) versus larger product effect size (e.g., d-prime >0.74) suggests the performance is “Satisfying”; worse performance accuracy (e.g., CA < 0.7) versus smaller product effect size (e.g., d-prime <0.74) suggests the performance is “Understandable”; worse performance accuracy (e.g., CA < 0.7) versus larger product effect size (e.g., d-prime >0.74) suggests the performance is “Unacceptable”.