Enhancing efficiency, reliability, and rigor in competency model analysis using natural language processing

Abstract

Background

Competency modeling is frequently used in higher education and workplace settings to inform a variety of learning and performance improvement programs. However, approaches commonly taken to modeling tasks can be very labor-intensive, and are vulnerable to perceptual and experience biases of raters.

Aims

The present study assesses the potential for natural language processing (NLP) to support competency-related tasks, by developing a baseline comparison of results generated by NLP to results generated by human raters.

Methods

Two raters separately conducted cross-walks for leadership competency models of graduate healthcare management programs from eight universities against a newly validated competency model from the National Center for Healthcare Leadership containing 28 competencies, to create 224 cross-walked pairs of “best matches”.

Results

Results indicated that the NLP model performed at least as accurately as human raters, who required a total of 16 work hours to complete, versus the NLP calculations which were nearly instantaneous.

Conclusion

Based on these findings, we conclude that NLP has substantial promise as a high-efficiency adjunct to human evaluations in competency cross-walks.