Can natural language processing or large language models replace human operators for pre-processing word and sentence-based free comments sensory evaluation data?

Abstract

The free comment (FC) method enables the collection of insights on products based on consumers' natural language. The primary drawback is the need for extensive data pre-processing. This study compared the results of three data pre-processing techniques applied to FC data related to the perception of six madeleines by two panels of 100 consumers: manual pre-processing by four human experts, automated pre-processing by an expert system, and automated pre-processing by the large language model ChatGPT. Two modes of data collection were used: responses only with words or short expressions (“FC words”), or responses based on complete sentences (“FC sentences”). Various indicators (number of words extracted, number of concepts retained, pre-processing time, level of repeatability/discrimination/stability of findings) were computed and compared between data collection modes and pre-processing techniques. It was shown that the automated systems performed correctly with FC words; however, they were less effective at extracting relevant words from FC sentences. The findings from statistical analyses following automated pre-processing were less repeatable and discriminative compared to those from the most proficient human operators. It was also demonstrated that, beyond the overall differences between products, the pre-processing of FC data can be a major source of non-reproducibility in findings, depending on the operators and the level of detail they consider when extracting information. Finally, the advantages and disadvantages of each pre-processing technique were summarized, along with several recommendations for pre-processing and analysing FC data at the appropriate level of granularity to draw robust conclusions.