Are we capturing individual differences? Evaluating the test-retest reliability of experimental tasks used to measure social cognitive abilities.

Abstract

Social cognitive skills are crucial for positive interpersonal relationships, health, and wellbeing and encompass both automatic and reflexive processes. To assess this myriad of skills, researchers have developed numerous experimental tasks that measure automatic imitation, emotion recognition, empathy, perspective taking, and intergroup bias and have used these to reveal important individual differences in social cognition. However, the very reason these tasks produce robust experimental effects - low between-participant variability - can make their use as correlational tools problematic. We performed an evaluation of test-retest reliability for common experimental tasks that measure social cognition. One-hundred and fifty participants completed the race-Implicit Association Test (r-IAT), Stimulus-Response Compatibility (SRC) task, Emotional Go/No-Go (eGNG) task, Dot Perspective-Taking (DPT) task, and State Affective Empathy (SAE) task, as well as the Interpersonal Reactivity Index (IRI) and indices of Explicit Bias (EB) across two sessions within 3 weeks. Estimates of test-retest reliability varied considerably between tasks and their indices: the eGNG task had good reliability (ICC = 0.63-0.69); the SAE task had moderate-to-good reliability (ICC = 0.56-0.77); the r-IAT had moderate reliability (ICC = 0.49); the DPT task had poor-to-good reliability (ICC = 0.24-0.60); and the SRC task had poor reliability (ICC = 0.09-0.29). The IRI had good-to-excellent reliability (ICC = 0.76-0.83) and EB had good reliability (ICC = 0.70-0.77). Experimental tasks of social cognition are used routinely to assess individual differences, but their suitability for this is rarely evaluated. Researchers investigating individual differences must assess the test-retest reliability of their measures.