Developing a single-session outcome measure using natural language processing on digital mental health transcripts

Abstract

Background

Current outcome measures in digital mental health lack granularity, especially for single-session interventions. This study aimed to address this by utilising natural language processing (NLP) methods to create a clear and relevant outcome measure. This paper describes the development of the Adult Session Wants and Needs Outcome Measure (Adult SWAN-OM), a novel outcome measure for the Qwell digital mental healthcare platform to understand service user (SU) needs engaging in single-session therapy (SST).

Methods

The research employs a multi-phased approach combining NLP methods with the typical stages of outcome measures development as follows: (1) assumption definition and validation with SUs and clinicians; (2) transcript theme extraction using the RoBERTa large language model (LLM) in conjunction with topic modelling to extract themes from 254 single-session transcripts from 192 SUs; (3) clinical item refinement focus group; (4) content validity with clinicians and SUs to improve the relevance and clarity of the items; and (5) outcome measure finalisation in a workshop held with clinicians to consolidate the final wording.

Results

Ninety-six potential wants and needs were generated and distilled into 12 measure items. The outcome measure was shown to be relevant and clear to both SUs and clinicians when used in the context of SST.

Conclusion

This study highlights the potential of combining NLP approaches with co-creation methods in single-session outcome measure development. We argue that the incorporation of clinical expertise and SU experience ensures the clarity and applicability of such measures and that this approach to capturing single-session wants and needs promises novel insights for supporting digital mental health interventions.