Development and evaluation of a 4M taxonomy from nursing home staff text messages using a fine-tuned generative language model.

Abstract

Objective: This study aimed to explore the utilization of a fine-tuned language model to extract expressions related to the Age-Friendly Health Systems 4M Framework (What Matters, Medication, Mentation, and Mobility) from nursing home worker text messages, deploy automated mapping of these expressions to a taxonomy, and explore the created expressions and relationships.

Materials and methods: The dataset included 21 357 text messages from healthcare workers in 12 Missouri nursing homes. A sample of 860 messages was annotated by clinical experts to form a "Gold Standard" dataset. Model performance was evaluated using classification metrics including Cohen's Kappa (κ), with κ ≥ 0.60 as the performance threshold. The selected model was fine-tuned. Extractions were clustered, labeled, and arranged into a structured taxonomy for exploration.

Results: The fine-tuned model demonstrated improved extraction of 4M content (κ = 0.73). Extractions were clustered and labeled, revealing large groups of expressions related to care preferences, medication adjustments, cognitive changes, and mobility issues.

Discussion: The preliminary development of the 4M model and 4M taxonomy enables knowledge extraction from clinical text messages and aids future development of a 4M ontology. Results compliment themes and findings in other 4M research.

Conclusion: This research underscores the need for consensus building in ontology creation and the role of language models in developing ontologies, while acknowledging their limitations in logical reasoning and ontological commitments. Further development and context expansion with expert involvement of a 4M ontology are necessary.