Leydi Johana Chaparro-Moreno, Hugo Gonzalez Villasanti, Laura M Justice, Jing Sun, Mary Beth Schmitt
{"title":"Accuracy of Automatic Processing of Speech-Language Pathologist and Child Talk During School-Based Therapy Sessions.","authors":"Leydi Johana Chaparro-Moreno, Hugo Gonzalez Villasanti, Laura M Justice, Jing Sun, Mary Beth Schmitt","doi":"10.1044/2024_JSLHR-23-00310","DOIUrl":null,"url":null,"abstract":"<p><strong>Purpose: </strong>This study examines the accuracy of Interaction Detection in Early Childhood Settings (IDEAS), a program that automatically transcribes audio files and estimates linguistic units relevant to speech-language therapy, including part-of-speech units that represent features of language complexity, such as adjectives and coordinating conjunctions.</p><p><strong>Method: </strong>Forty-five video-recorded speech-language therapy sessions involving 27 speech-language pathologists (SLPs) and 56 children were used. The <i>F</i> measure determines the accuracy of IDEAS diarization (i.e., speech segmentation and speaker classification). Two additional evaluation metrics, namely, median absolute relative error and correlation, indicate the accuracy of IDEAS for the estimation of linguistic units as compared with two conditions, namely, Oracle (manual diarization) and Voice Type Classifier (existing diarizer with acceptable accuracy).</p><p><strong>Results: </strong>The high <i>F</i> measure for SLP talk data suggests high accuracy of IDEAS diarization for SLP talk but less so for child talk. These differences are reflected in the accuracy of IDEAS linguistic unit estimates. IDEAS median absolute relative error and correlation values for nine of the 10 SLP linguistic unit estimates meet the accuracy criteria, but none of the child linguistic unit estimates meet these criteria. The type of linguistic units also affects IDEAS accuracy.</p><p><strong>Conclusions: </strong>IDEAS was tailored to educational settings to automatically convert audio recordings into text and to provide linguistic unit estimates in speech-language therapy sessions and classroom settings. Although not perfect, IDEAS is reliable in automatically capturing and returning linguistic units, especially in SLP talk, that are relevant in research and practice. The tool offers a way to automatically measure SLP talk in clinical settings, which will support research seeking to understand how SLP talk influences children's language growth.</p>","PeriodicalId":51254,"journal":{"name":"Journal of Speech Language and Hearing Research","volume":null,"pages":null},"PeriodicalIF":2.2000,"publicationDate":"2024-08-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Speech Language and Hearing Research","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1044/2024_JSLHR-23-00310","RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2024/7/17 0:00:00","PubModel":"Epub","JCR":"Q1","JCRName":"AUDIOLOGY & SPEECH-LANGUAGE PATHOLOGY","Score":null,"Total":0}
引用次数: 0
Abstract
Purpose: This study examines the accuracy of Interaction Detection in Early Childhood Settings (IDEAS), a program that automatically transcribes audio files and estimates linguistic units relevant to speech-language therapy, including part-of-speech units that represent features of language complexity, such as adjectives and coordinating conjunctions.
Method: Forty-five video-recorded speech-language therapy sessions involving 27 speech-language pathologists (SLPs) and 56 children were used. The F measure determines the accuracy of IDEAS diarization (i.e., speech segmentation and speaker classification). Two additional evaluation metrics, namely, median absolute relative error and correlation, indicate the accuracy of IDEAS for the estimation of linguistic units as compared with two conditions, namely, Oracle (manual diarization) and Voice Type Classifier (existing diarizer with acceptable accuracy).
Results: The high F measure for SLP talk data suggests high accuracy of IDEAS diarization for SLP talk but less so for child talk. These differences are reflected in the accuracy of IDEAS linguistic unit estimates. IDEAS median absolute relative error and correlation values for nine of the 10 SLP linguistic unit estimates meet the accuracy criteria, but none of the child linguistic unit estimates meet these criteria. The type of linguistic units also affects IDEAS accuracy.
Conclusions: IDEAS was tailored to educational settings to automatically convert audio recordings into text and to provide linguistic unit estimates in speech-language therapy sessions and classroom settings. Although not perfect, IDEAS is reliable in automatically capturing and returning linguistic units, especially in SLP talk, that are relevant in research and practice. The tool offers a way to automatically measure SLP talk in clinical settings, which will support research seeking to understand how SLP talk influences children's language growth.
期刊介绍:
Mission: JSLHR publishes peer-reviewed research and other scholarly articles on the normal and disordered processes in speech, language, hearing, and related areas such as cognition, oral-motor function, and swallowing. The journal is an international outlet for both basic research on communication processes and clinical research pertaining to screening, diagnosis, and management of communication disorders as well as the etiologies and characteristics of these disorders. JSLHR seeks to advance evidence-based practice by disseminating the results of new studies as well as providing a forum for critical reviews and meta-analyses of previously published work.
Scope: The broad field of communication sciences and disorders, including speech production and perception; anatomy and physiology of speech and voice; genetics, biomechanics, and other basic sciences pertaining to human communication; mastication and swallowing; speech disorders; voice disorders; development of speech, language, or hearing in children; normal language processes; language disorders; disorders of hearing and balance; psychoacoustics; and anatomy and physiology of hearing.