{"title":"评估 ChatGPT 生成的文本摘要的多指标方法","authors":"Jonas Benedikt Arnold;Dominik Hörauf","doi":"10.1109/EMR.2024.3381176","DOIUrl":null,"url":null,"abstract":"This article investigates the summarization capabilities of ChatGPT, a language model seen as effectively shortening texts, employing a hypothesis-generating and explorative approach. Using a specific prompt, the study examines the expected lengths of generated summaries across various input word counts (IWC). A shortening ratio is introduced to describe these relationships, with identified dependencies on IWCs between 100 and 400 words. The study also explores coherence comparisons, highlighting that the ChatGPT-generated text is often evaluated as more coherent than the original. The article introduces a multimetric approach for the evaluation and discusses dependencies of best case summaries on different input word counts, providing insights into the model's performance characteristics.","PeriodicalId":35585,"journal":{"name":"IEEE Engineering Management Review","volume":null,"pages":null},"PeriodicalIF":0.0000,"publicationDate":"2024-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"A Multimetric Approach for Evaluation of ChatGPT-Generated Text Summaries\",\"authors\":\"Jonas Benedikt Arnold;Dominik Hörauf\",\"doi\":\"10.1109/EMR.2024.3381176\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"This article investigates the summarization capabilities of ChatGPT, a language model seen as effectively shortening texts, employing a hypothesis-generating and explorative approach. Using a specific prompt, the study examines the expected lengths of generated summaries across various input word counts (IWC). A shortening ratio is introduced to describe these relationships, with identified dependencies on IWCs between 100 and 400 words. The study also explores coherence comparisons, highlighting that the ChatGPT-generated text is often evaluated as more coherent than the original. The article introduces a multimetric approach for the evaluation and discusses dependencies of best case summaries on different input word counts, providing insights into the model's performance characteristics.\",\"PeriodicalId\":35585,\"journal\":{\"name\":\"IEEE Engineering Management Review\",\"volume\":null,\"pages\":null},\"PeriodicalIF\":0.0000,\"publicationDate\":\"2024-06-01\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"IEEE Engineering Management Review\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://ieeexplore.ieee.org/document/10601574/\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q1\",\"JCRName\":\"Business, Management and Accounting\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"IEEE Engineering Management Review","FirstCategoryId":"1085","ListUrlMain":"https://ieeexplore.ieee.org/document/10601574/","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"Business, Management and Accounting","Score":null,"Total":0}
A Multimetric Approach for Evaluation of ChatGPT-Generated Text Summaries
This article investigates the summarization capabilities of ChatGPT, a language model seen as effectively shortening texts, employing a hypothesis-generating and explorative approach. Using a specific prompt, the study examines the expected lengths of generated summaries across various input word counts (IWC). A shortening ratio is introduced to describe these relationships, with identified dependencies on IWCs between 100 and 400 words. The study also explores coherence comparisons, highlighting that the ChatGPT-generated text is often evaluated as more coherent than the original. The article introduces a multimetric approach for the evaluation and discusses dependencies of best case summaries on different input word counts, providing insights into the model's performance characteristics.
期刊介绍：
Reprints articles from other publications of significant interest to members. The papers are aimed at those engaged in managing research, development, or engineering activities. Reprints make it possible for the readers to receive the best of today"s literature without having to subscribe to and read other periodicals.
京公网安备 11010802042870号
京ICP备2023020795号-1
分享
求助内容：
应助结果提醒方式：
扫码关注我们
