Words of Warning: A Randomized Study of the Impact of Assorted Warning Letters on Academic Probation Students

Abstract

Objective: Superiority experiments supply a critical lens to probe higher education interventions because they can dismantle the collective effect of a program and identify the relative contribution of the core components. However, such lines of inquiry are generally absent in higher education literature. We draw on experimental data to probe the dominant mechanisms of a simple but promising academic probation intervention. Methods: We used a 2 × 2 factorial experiment designed to unpack the effects of an academic probation intervention and identify the dominant mechanism. Within this framework, 500 academic probation students were assigned to one of four treatment conditions or a control condition. Treatment groups were assigned to receive notification of academic probation status by either certified or regular mail (factor A) that contained either a standard or embellished letter (factor B). Results: We found that students who received embellished warning letters significantly improved subsequent academic performance. The impact on performance was 0.35 grade points when notification sent by regular mail and amplified to 0.48 grade points when sent via certified mail. The routine notification communication message did not differ from the treatment as usual method regardless of mail delivery method. Conclusions: These findings suggest the availability of low-cost, scalable interventions are available and can be identified to positively impact at-risk students’ academic achievement.