Sentence-BERT Distinguishes Good and Bad Essays in Cross-prompt Automated Essay Scoring

Abstract

Automated Essay Scoring (AES) refers to a set of processes that automatically assigns grades to student-written essays with machine learning models. Existing AES models are mostly trained prompt-specifically with supervised learning, which requires the essay prompt to be accessible to the system vendor at the time of model training. However, essay prompts for high-stakes testing should usually be kept confidential before the test date, which demands the model to be cross-promptly trainable with pre-scored essay data already in hands. Document embeddings obtained from pretrained language models such as Sentence-BERT (sbert) are primarily expected to represent the semantic content of the text. We hypothesize SBERT embeddings also contain assessment-relevant elements that are extractable by document embedding decomposition through Principal Component Analysis (PCA) enhanced with Normalized Discounted Cumulative Gain (nDCG) measurement. The identified evaluative elements in the entire embedding space of the source essays are then cross-promptly transferred to the target essays written on different prompts for binary clustering task of dividing high/low-scored groups. The result implies non-finetuned SBERT already contains evaluative elements to distinguish good and bad essays.