Minding the (achievement) gap

Abstract

Fundamentally, Reeves and I agree about the importance of boys’ educational under-achievement and the need to openly discuss and address it. I emphasize that when boys fail to thrive in school, it has downstream consequences not only for their own lives but for our nation's economic growth. Further, boys’ success need not come at the expense of girls’ success. This is not zero-sum; we all benefit when children reach their potential.

I must push back against the narrative that these gaps are being hidden from public view and researchers have not taken them seriously. Nothing could be further from the truth.

An old adage in business circles is “what gets measured gets done.” Indeed, a pillar of the school accountability movement is providing clear, disaggregated data at the state, district, and local levels so that parents and community members can better understand and monitor student achievement.

Reeves argues that we often choose to ignore gender gaps, stating “At the extreme, gender-neutrality veers into gender-blind approach: some school districts, for example, do not even routinely track differences in outcomes by gender.” Such actions would be in violation of the law. Under the Every Student Succeeds Act (ESSA; 2015), states must publicly report school accountability data in a manner must be disaggregated by a number of categories, including gender.

In his Point in this exchange (and also in his book, Reeves, 2022), Reeves describes results from a randomized experiment aimed at improving academic performance among college students. In the experiment, students were offered academic support services, financial incentives ranging from $1000 to $5000 for attaining high grades, or both. While the treatment improved women's grades and academic standing, it had no effect on men's outcomes (Angrist et al., 2009). Reeves goes on to conclude, “Josh Angrist and co-authors wrote in 2009: ‘These gender differences in the response to incentives and services constitute an important area for further study.’ They do indeed. But as far as I can see, nobody has heeded this call.”

On the contrary, many have heeded the call. The Angrist et al. (2009) article has more than 700 citations, many of these citations came from studies that have attempted to better understand gender gaps and how to design policies to improve outcomes for males and females. (Indeed, there is a large literature on this topic; see Croson and Gneezy, 2009, for a review.) Many of these have been randomized-controlled trials, generally considered to be the gold standard of research. Curiously, none of these are cited in Reeves's work. Below I summarize primarily the work on higher education relevant to the Angrist et al. (2009) quote, but I also note that there is also substantial work on pre-K, elementary and secondary schools, and labor market outcomes.

When it comes to financial incentives for performance in higher education, the literature on gender effects is far from settled. Two prominent randomized experiments offered financial incentives to college students and found heterogeneous impacts by students’ achievement levels, which may be correlated with gender. In an experiment in the Netherlands, Leuven et al. (2010) found that financial incentives improve the likelihood that college students pass all first-year requirements within a year, but only among high-ability students. Incentives were found to have a negative impact on achievement of low-ability students. In an experiment in an Italian university, students were offered small (250 euros) or large (700 euros) monetary prizes for being among the highest performers on credits earned and examination grades in the first-year undergraduate class (De Paola et al., 2012). The prizes improved student performance, though the differences across prize levels were quite modest with a 0.19 standard deviation increase in response to the large prize and a 0.16 standard deviation increase in response to the small one. Similar to the results from the Netherlands, the impacts were concentrated among high-ability students, with no impact among low-ability students. Consistent with differences in achievement levels, the Italian study found suggestive evidence of larger point estimates for females (De Paola et al., 2012).

Performance-based scholarships, a different type of financial incentive, also have inconsistent impacts by gender. Mayer et al. (2015) found that modest scholarships appear to improve credits earned and enrollment more among males than females (though they do not test for statistical differences across groups). By contrast, Cohodes and Goodman (2014) found no differences in impact of merit aid on enrollment, college quality, or completion by gender. Dynarski et al. (2021) studied the impact of a randomized experiment that provided an early commitment of free tuition at a selective flagship university. Note the intervention did not increase aid, but instead provided information and resolved uncertainty. Targeted students were more likely to apply, be admitted, and enroll than the control group; while they find larger point estimates for women than men for admission and enrollment, the gender differences are not quite statistically significant (p = 0.15).

Access to academic support also has inconsistent impacts by gender. As previously mentioned, Angrist et al. (2009) found that academic support services helped women but not men. By contrast, Bettinger and Baker (2014) evaluated a randomized experiment in which college students were provided coaches who helped treated students develop clear, long-term goals and connect their daily activities to those goals, e.g., through improving skills in time-management, self-advocacy, and studying. The impacts on college persistence was positive, with the impact greater for males than for females, through 24 months post-assignment.

Building on these and other studies, Clark et al. (2020) conducted a series of fascinating experiments designed to better understand how to structure interventions to improve college outcomes, with special attention to gender differences. In particular, they compared the impacts of task-based goals (i.e., asking students to set a goal for the number of practice quizzes they would take over the course of the semester) against performance-based goals (i.e., asking students to set a goal final letter grade).

Clark et al. (2020) drew insights from the prior literature which suggests that males are expected to be more responsive to task-based goals for a variety of reasons. Males have been shown on average to have less self-control in educational environments than females do, suggesting that interventions aimed at improving self-control may have a stronger impact on them (Duckworth & Seligman, 2005; Duckworth et al., 2015). Further, goal-setting interventions appear to be more successful for males overall and for students who think more about the present than the future (i.e., are “present-biased,” a trait also more common among males). In addition, there are gender differences in responses to competition (Gneezy et al., 2003). For example, Azmat et al. (2016) have shown that girls tend to outperform boys on tests, but their advantage declines the stakes on a test become higher.

Consistent with the theoretical framework, the experiments conducted by Clark et al. (2020) found that males are more responsive to setting task-based goals and the intervention in turn has a positive impact on their final course grades. By contrast, females neither complete more tasks nor improve their course grades when assigned to set task-based goals. Performance-based goals had no effect on males or females.

To be sure, the literature is largely unsettled. In some cases, we still do not have solid evidence on whether interventions vary by gender. We surely do not know enough about why the impacts of some interventions vary by gender. But this reflects the difficulty of the challenge, and not a lack of serious attention to the problem.