British Journal of Mathematical & Statistical Psychology最新文献

Reliability is crucial in psychometrics, reflecting the extent to which a measurement instrument can discriminate between individuals or items. While classical test theory and intraclass correlation coefficients are well-established for quantitative scales, estimating reliability for binary outcomes presents unique challenges due to their discrete nature. This paper reviews and links three major approaches to estimate reliability for single ratings on binary scales: the normal approximation approach, kappa coefficients, and the latent variable approach, which enables estimation at both latent and manifest scale levels. We clarify their conceptual relationships, show conditions for asymptotical equivalence, and evaluate their performance across two common study designs, repeatability and reproducibility studies. Then, we extend the Bayesian Dirichlet-multinomial method for estimating kappa coefficients to settings with more than two replicates, without requiring Bayesian software. Additionally, we introduce a Bayesian method to estimate manifest scale reliability from latent scale reliability that can be implemented in standard Bayesian software. A simulation study compares the statistical properties of the three major approaches across Bayesian and frequentist frameworks. Overall, the normal approximation approach performed poorly, and the frequentist approach was unreliable due to singularity issues. The findings offer further refined practical recommendations.

Reliability evaluation is critical in fields such as psychology and medicine to ensure accurate diagnosis and effective treatment management. When participants are evaluated by the same raters, a two-way ANOVA model is suitable to model the data, with the intraclass correlation coefficient (ICC) serving as the reliability metric. In these domains, the ICC for agreement (ICCa) is commonly used, as the values of the measurements themselves are of interest. Designing such reliability studies requires determining the sample size of participants and raters for the ICCa. Although procedures for sample size determination exist based on the expected width of the confidence interval for the ICCa, there is limited work on hypothesis testing. This paper addresses this gap by proposing procedures to ensure sufficient power to statistically test whether the ICCa exceeds a predetermined value, utilizing confidence intervals for the ICCa. We compared the available confidence interval methods for the ICCa and proposed sample size procedures using the lower confidence limit of the best performing methods. These procedures were evaluated considering the empirical power of the hypothesis test under various parameter configurations. Furthermore, these procedures are implemented in an interactive R shiny app, freely available to researchers for determining sample sizes.

This paper introduces two new Item Response Theory (IRT) models, based on the Generalized Extreme Value (GEV) distribution. These new models have asymmetric item characteristic curves (ICC) which have drawn growing interest, as they may better model actual item response behaviours in specific scenarios. The analysis of the models is carried out using a Bayesian approach, and their properties are examined and discussed. The validity of the models is verified by means of extensive simulation studies to evaluate the sensitivity of the model to the choice of priors on the new item parameter introduced, the accuracy of the parameters' recovery, as well as an assessment of the capacity of model comparison criteria to choose the best model against other IRT models. The new models are exemplified using real data from two mathematics tests, one applied in Peruvian public schools and another one administered to incoming university students in Chile. In both cases, the proposed models showed to be a promising alternative to asymmetric IRT models, offering new insights into item response modelling.

In knowledge structure theory (KST) framework, this study evaluates the reliability of knowledge state estimation by introducing two key measures: the expected accuracy rate and the expected discrepancy. The accuracy rate quantifies the likelihood that the estimated knowledge state aligns with the true state, while the expected discrepancy measures the average deviation when misclassification occurs. To support the theoretical framework, we provide an in-depth discussion of these indices, supplemented by two simulation studies and an empirical example. The simulation results reveal a trade-off between the number of items and the size of the knowledge structure. Specifically, smaller structures exhibit consistent accuracy across different error levels, while larger structures show increasing discrepancies as error rates rise. Nevertheless, accuracy improves with a greater number of items in larger structures, mitigating the impact of errors. Additionally, the expected discrepancy analysis shows that when misclassification occurs, the estimated state is generally close to the true one, minimizing the effect of errors in the assessment. Finally, an empirical application using real assessment data demonstrates the practical relevance of the proposed measures. This suggests that KST-based assessments provide reliable and meaningful diagnostic information, highlighting their potential for use in educational and psychological testing.

The stationary autoregressive model forms an important base of time-series analysis in today's psychology research. Diverse nonstationary extensions of this model are developed to capture different types of changing temporal dynamics. However, researchers do not always have a solid theoretical base to rely on for deciding a-priori which of these nonstationary models is the most appropriate for a given time-series. In this case, correct model selection becomes a crucial step to ensure an accurate understanding of the temporal dynamics. This study consists of two main parts. First, with a simulation study, we investigated the performance of in-sample (information criteria) and out-of-sample (cross-validation, out-of-sample prediction) model selection techniques in identifying six different univariate nonstationary processes. We found that the Bayesian information criteria (BIC) has an overall optimal performance whereas other techniques' performance depends largely on the time-series' length. Then, we re-analysed a 239-day-long time-series of positive and negative affect to illustrate the model selection process. Combining the simulation results and practical considerations from the empirical analysis, we argue that model selection for nonstationary time-series should not completely rely on data-driven approaches. Instead, more theory-driven approaches where researchers actively integrate their qualitative understanding will inform the data-driven approaches.

Reinforcement learning (RL) powers the engine of adaptive learning systems which recommend customized learning materials to individual learners in their varying learning states to optimize learning effectiveness. However, some argue that only improving learning effectiveness may be insufficient, particularly if it overly extends learning efforts and requires additional time to work on the recommended materials. Learners with different amounts of prior knowledge consume different amounts of time on the same material. Therefore, designers should consider both the usefulness of the material and the time dedicated to making sense of the materials by individual learners with a specific amount of prior knowledge. To fill this gap, this study proposes a RL-based adaptive learning system wherein reward is improved by considering both factors. We then conducted Monte Carlo simulation studies to verify the effects of the improved reward and uncover the mechanisms for RL recommendation strategies. Results show that the improved reward reduces learners' learning duration substantially due to interpretable recommendation strategies, which results in growing learning efficiency for learners with varying prior knowledge.

The basic local independence model (BLIM) is appropriate in situations where populations do not differ in the probabilities of the knowledge states and the probabilities of careless errors and lucky guesses of the items. In some situations, this is not the case. This work introduces the multiple observed classification local independence model (MOCLIM), which extends the BLIM by allowing the above probabilities to vary across populations. In the MOCLIM, each individual is characterized by proficiency, careless and guessing classes, which are observed and determine the probabilities of knowledge states, careless errors and lucky guesses of a population. Given a particular class type (proficiency, careless, or guessing), the probabilities are the same for populations with the same class but may vary between populations with different classes. Algorithms for maximum likelihood estimation of the MOCLIM parameters are provided. The results of a simulation study suggest that the true parameter values are well recovered by the estimation algorithm and that the true model can be uncovered by comparing the goodness-of-fit of alternative models. The results of an empirical application to data from Raven-like matrices suggest that the MOCLIM effectively discriminates between situations where group differences are expected and those where they are not.

The Bayes factor, the data-based updating factor of the prior to posterior odds of two hypotheses, is a natural measure of statistical evidence for one hypothesis over the other. We show how Bayes factors can also be used for parameter estimation. The key idea is to consider the Bayes factor as a function of the parameter value under the null hypothesis. This 'support curve' is inverted to obtain point estimates ('maximum evidence estimates') and interval estimates ('support intervals'), similar to how p-value functions are inverted to obtain point estimates and confidence intervals. This provides data analysts with a unified inference framework as Bayes factors (for any tested parameter value), support intervals (at any level), and point estimates can be easily read off from a plot of the support curve. This approach shares similarities but is also distinct from conventional Bayesian and frequentist approaches: It uses the Bayesian evidence calculus, but without synthesizing data and prior, and it defines statistical evidence in terms of (integrated) likelihood ratios, but also includes a natural way for dealing with nuisance parameters. Applications to meta-analysis, replication studies and logistic regression illustrate how our framework is of practical value for making quantitative inferences.

Intensive longitudinal data are often found to be non-stationary, namely, showing changes in statistical properties, such as means and variance-covariance structures, over time. One way to accommodate non-stationarity is to specify key parameters that show over-time changes as time-varying parameters (TVPs). However, the nature and dynamics of TVPs may themselves be heterogeneous across time, contexts, developmental stages, individuals and as related to other biopsychosocial-cultural influences. We propose an outlier detection method designed to facilitate the detection of critical shifts in any differentiable linear and non-linear dynamic functions, including dynamic functions for TVPs. This approach can be readily applied to various data scenarios, including single-subject and multisubject, univariate and multivariate processes, as well as with and without latent variables. We demonstrate the utility and performance of this approach with three sets of simulation studies and an empirical illustration using facial electromyography data from a laboratory emotion induction study.