Reflections on a mentored group peer review process

Abstract

Peer review, an established part of research practice, is intended to ensure quality and engender trust among researchers. Reviewers with appropriate expertise evaluate a manuscript to ensure scholarly practices and engage in productive dialogue with the authors. However, significant concerns with peer review (Ware, 2008) have sparked proposals to improve the process. Biases can manifest through prestige, nationality, gender, and content (Lee et al., 2013). Because the peer review process is primarily maintained by unpaid labor, it burdens overworked scholars, leading to fewer and fewer willing peer reviewers and, at times, rushed reviews (Dance, 2023; Flaherty, 2022). Delays in peer review are an issue for junior faculty seeking promotion and graduate students entering the job market (Dance, 2023). Rushed peer reviews have been shown to miss errors, leading to erroneous publications that erode trust in the scientific community (Campbell, 2024; Lowe, 2010).

Previous JEE editorials have addressed the challenges of establishing a new peer review culture in STEM (Benson, 2019; Knight & Main, 2024) and the positive influences of the JEE Mentored Review Program on the identity of mentees (Jensen et al., 2021). Other solutions have been proposed to promote more equitable and efficient peer review. Some claim that crowdsourcing peer reviews using discipline-specific online forums has been fast and effective (List, 2017); however, this approach has yet to be widely explored. Artificial intelligence has also been used to aid the review process by potentially improving the quality of reviews and addressing the lack of reviewers; however, this remedy raises substantial concerns about bias, reliability, and appropriate use of data (Hosseini & Horbach, 2023).

Here, we provide reflections from a mentored peer review process within a single engineering education research group. We assert that engaging in this process offers a shared learning experience where emerging scholars can learn about an essential research practice. Furthermore, it has the potential to grow the number of qualified reviewers, improve paper quality, and increase reviewers' academic reading and writing confidence—all while providing quality feedback in a specific review.

This mentored review process can be viewed through a community of practice lens (Wenger, 1998) where novices (e.g., graduate students new to manuscript writing and reviewing) engage as legitimate peripheral participants (Lave & Wenger, 1991) in authentic practice to interact with more central participants and learn the sociotechnical practices of the community. The designers of the JEE Mentored Peer Review Program take this approach, giving junior faculty members experience by matching them with experienced mentors across three structured peer reviews (Jensen et al., 2021). Mentees within the program reported that it had a positive impact on peer review-specific self-confidence and helped develop interpersonal networks.

With these benefits in mind, we engaged in a mentored group peer review process for graduate students, postdocs, and the faculty advisor within one research group. Unlike many mentorship pairings, a research group is a pre-existing community with strong social ties where members regularly interact and have a shared domain of interest. Group interaction allows for the formation of group practices where shared meaning is created, here in the review of a manuscript (Stahl, 2017). These group practices are part of a “mutually supportive mechanism” (Medina & Stahl, 2021, p. 200) for learning at three stages: individual, group, and the larger community. A mentored group peer review allows individuals to participate in group practices, enabling their growth as disciplinary scholars.

Below, we describe our group peer review process, reflections, and recommendations.

Figure 1 shows the steps and timeline of our mentored group review process.

After consulting with the journal editors, the faculty advisor assigned a postdoc as the facilitator to lead the review. Six other group members, with a shared focus in engineering education research, but with various levels of academic experience and diverse disciplinary backgrounds (chemical engineering, computer science, engineering design, and mechanical engineering), participated in this process. At a group meeting, the facilitator shared the paper and outlined the JEE journal review criteria, communicating that within the bounds of the criteria, each person could craft their feedback as they wished. Participants then sent their individual comments to the facilitator, who reviewed and synthesized the group's main ideas. To encourage feedback, transparency, and discussion, the facilitator led a group discussion based on the review criteria and individual comments. The faculty advisor was present during this discussion to answer questions about the review process and/or offer additional thoughts unrelated to their review of the paper. After the group meeting, the facilitator composed a review draft. Following feedback incorporation from other participants and a quality check from the faculty advisor, the facilitator submitted the review. A reflective discussion and opportunity to provide written feedback concluded this process.

Table 1 describes the potential benefits for the different groups involved in this review process.

By engaging in productive dialogue with experienced reviewers, novices felt that their understanding of the review process, as well as their confidence around critical reading and academic writing, increased. Specifically, some novices benefited from group affirmations when discussing points of confusion within the manuscript that they reviewed; novices who were initially hesitant about the legitimacy of their critiques developed confidence in their ability to identify potential gaps or flaws in the argument.

Experienced reviewers gained a structured opportunity to practice their skills of reviewing a journal manuscript and mentor novice reviewers. Experienced reviewers felt this context could inform future mentoring interactions. Both novice and experienced reviewers appreciated insights from additional perspectives on the manuscript. This not only allowed for exposure to different viewpoints, but also presented possible perspectives that peer reviewers may take when reviewing papers, which will inform group members' future writing.

The mentored group peer review process supported the faculty advisor's goal of developing the mentees’ skills related to peer review and the publishing process. Additionally, the faculty mentor felt that engaging in the process reinforced a shared group culture that valued sensemaking, respect, and collegiality. Finally, due to the process's flexibility, the individual strengths and experiences of mentees were highlighted.

For the journal, this process offers an opportunity to invest in the development of qualified peer reviewers. Furthermore, participants in this process will likely become better writers and thus submit higher quality manuscripts to the journal in the future, as peer review has been observed to promote improved writing skills (Finkenstaedt-Quinn et al., 2021; Lundstrom & Baker, 2009).

Unless mentorship of graduate students explicitly provides opportunities for professional development, students sometimes find that their doctoral work leaves them unprepared for the responsibilities of academic roles (Renbarger et al., 2021). Engaging in this activity provided opportunities for both novice and experienced reviewers to build skills essential to becoming successful scholars.

We believe this process provides value across scholarly disciplines as a professional development tool for graduate students and postdocs by engaging in authentic practice. In the following sections, we describe our recommendations for research groups interested in pursuing group peer review.