Collaborative survey construction for national data collection: Coordination, negotiation, and delivery

Abstract

This research-to-practice full paper describes the deliberate and arduous process we recently went through to develop a national survey to study the non-cognitive traits of undergraduate engineering and computing students. The goal of this survey is to characterize student profiles in order to develop and examine particular interventions to guide students toward success in engineering and computing majors. This survey measures non-cognitive attributes including personality, sense of belonging, engineering or computing identity, study skills, well-being, and a variety of other constructs that are not routinely measured in engineering populations nor integrated into admission decisions, advising processes, or academic curricula. Prior research indicates that these non-cognitive attributes are important for students’ academic success and retention. However, no studies have examined a comprehensive set of non-cognitive traits holistically to understand how they influence student success. This collaborative project, funded by three linked NSF grants, merges the interests of researchers at three campuses in understanding and supporting students with varied non-cognitive profiles. As part of this research, we negotiated the content of a national survey, suitable for use on our own campuses as well as with other national partners, to probe more than a dozen constructs collectively describing student non-cognitive attributes. The construction of the survey itself was non-trivial, and involved significant negotiations among the researchers including initial collection of instruments with validity evidence to serve as a basis for discussion; an in-person kick-off meeting; multiple follow-up teleconferences; multiple rounds of inclusion/exclusion decisions based upon mutually-agreed upon guiding principles; pilot survey testing; pilot data evaluations such as exploratory factor analysis; and final decisions about instruments/items to include in the final version, all while considering survey length, distribution channels, and key IRB concerns. This paper details the 10-month effort to construct a survey that meets the research needs and intellectual curiosity of partners at three diverse campuses. In this process, we had to balance the different institutional contexts of the funded partner sites while also maintaining flexibility for national distribution. The deliberate processes we used may serve as a template for future survey creation, starting from constructs of interest, to selection of specific instruments (or sub-scales thereof), and factor analysis to consider further down-selection of individual items to include in the final survey. The outcomes of this paper may serve the engineering education community by highlighting previously undocumented processes in collaborative survey construction that introduce intellectual complexity or time delays into the development timeline.