Kimberly D Becker, Eleanor G Wu, Jonathan G Westman, Meredith R Boyd, Karen Guan, Davielle Lakind, Wendy Chu, Kendra S Knudsen, W Joshua Bradley, Alayna L Park, Tara Kenworthy LaMarca, Emily Lang, Bruce F Chorpita
{"title":"The Interrater Reliability of a Coding System for Measuring Mental Health Professionals' Decisions and Actions.","authors":"Kimberly D Becker, Eleanor G Wu, Jonathan G Westman, Meredith R Boyd, Karen Guan, Davielle Lakind, Wendy Chu, Kendra S Knudsen, W Joshua Bradley, Alayna L Park, Tara Kenworthy LaMarca, Emily Lang, Bruce F Chorpita","doi":"10.1080/15374416.2024.2384027","DOIUrl":null,"url":null,"abstract":"<p><strong>Objective: </strong>The clinical decisions and actions of evidence-based practice in psychology (EBPP) are largely underspecified and poorly understood, in part due to the lack of measurement methods. We tested the reliability of a behavioral coding system that characterizes a flow of interrelated activities that includes problem detection and prioritization, intervention selection and implementation, and review of intervention integrity and impact.</p><p><strong>Method: </strong>The context included two publicly funded youth mental health service organizations located in geographically distinct and underresourced communities in the U.S. where service inequities are common. We sampled 84 digitally recorded and transcribed supervision events that included a sample of professionals who were mostly women (93.02%) and BIPOC (86.04%) whose self-reported race/ethnicity matched the youth populations they served. We coded these events for activities (e.g., considering) and their predicate content (i.e., problems or practices) and examined reliability of these codes applied to excerpts (i.e., small contiguous units of dialogue) as well as to complete events.</p><p><strong>Results: </strong>Interrater reliability estimates showed that, overall, coders reliably rated the occurrence and extensiveness of activities and content. Excerpt coding was generally more reliable than event coding. However, mathematical aggregation of excerpt coding offered a superior method for estimating event codes reliably, reducing individual subjectivity while providing event level synthesis of activities that are grounded in excerpt level details.</p><p><strong>Conclusions: </strong>The assessment of clinical decisions and actions has the potential to unpack the black box of EBPP, with different methods best suited to different research questions and resource considerations.</p>","PeriodicalId":48350,"journal":{"name":"Journal of Clinical Child and Adolescent Psychology","volume":null,"pages":null},"PeriodicalIF":4.2000,"publicationDate":"2024-08-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Clinical Child and Adolescent Psychology","FirstCategoryId":"102","ListUrlMain":"https://doi.org/10.1080/15374416.2024.2384027","RegionNum":1,"RegionCategory":"心理学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"PSYCHOLOGY, CLINICAL","Score":null,"Total":0}
引用次数: 0
Abstract
Objective: The clinical decisions and actions of evidence-based practice in psychology (EBPP) are largely underspecified and poorly understood, in part due to the lack of measurement methods. We tested the reliability of a behavioral coding system that characterizes a flow of interrelated activities that includes problem detection and prioritization, intervention selection and implementation, and review of intervention integrity and impact.
Method: The context included two publicly funded youth mental health service organizations located in geographically distinct and underresourced communities in the U.S. where service inequities are common. We sampled 84 digitally recorded and transcribed supervision events that included a sample of professionals who were mostly women (93.02%) and BIPOC (86.04%) whose self-reported race/ethnicity matched the youth populations they served. We coded these events for activities (e.g., considering) and their predicate content (i.e., problems or practices) and examined reliability of these codes applied to excerpts (i.e., small contiguous units of dialogue) as well as to complete events.
Results: Interrater reliability estimates showed that, overall, coders reliably rated the occurrence and extensiveness of activities and content. Excerpt coding was generally more reliable than event coding. However, mathematical aggregation of excerpt coding offered a superior method for estimating event codes reliably, reducing individual subjectivity while providing event level synthesis of activities that are grounded in excerpt level details.
Conclusions: The assessment of clinical decisions and actions has the potential to unpack the black box of EBPP, with different methods best suited to different research questions and resource considerations.
期刊介绍:
The Journal of Clinical Child and Adolescent Psychology (JCCAP) is the official journal for the Society of Clinical Child and Adolescent Psychology, American Psychological Association. It publishes original contributions on the following topics: (a) the development and evaluation of assessment and intervention techniques for use with clinical child and adolescent populations; (b) the development and maintenance of clinical child and adolescent problems; (c) cross-cultural and sociodemographic issues that have a clear bearing on clinical child and adolescent psychology in terms of theory, research, or practice; and (d) training and professional practice in clinical child and adolescent psychology, as well as child advocacy.