A Methodology for Determining and Validating Latent Factor Dimensionality of Complex Multi-Factor Science Constructs Measuring Knowledge-In-Use

ABSTRACT Deep science understanding is reflected in students’ ability to use content and skills when making sense of the world. Assessing deep understanding requires measuring complex constructs that combine elements of content and skills. To develop valid measures of complex constructs, we need to understand how their theoretical dimensionality, reflected in the integration of content and skills, is manifested in practice. This work is developed in the context of the Framework for K-12 Science Education and Next-Generation Science Standards (NGSS). We introduce a methodology that describes steps for creating a theoretical validity argument for measuring complex NGSS constructs, designing operational assessments based on this argument, and obtaining empirical evidence for the validity of the argument and assessments, focusing on how theoretically suggested dimensionality of NGSS constructs is manifested in practice. Results have implications for developing valid NGSS assessments and reporting student progress on high-stakes and diagnostic evaluation.