Cl2sum: abstractive summarization via contrastive prompt constructed by LLMs hallucination

Abstract

The rise of Large Language Models (LLMs) has further led to the development of text summarization techniques and also brought more attention to the problem of hallucination in the research of text summarization. Existing work in current text summarization research based on LLMs typically uses In-Context Learning (ICL) to supply accurate (document-summary) pairs of samples to the model, thus allowing the model to be more explicit in predicting the target. However, in this way, models can only determine what to do, without explicitly prohibiting what models cannot do. It is highly likely to lead to increased hallucinations due to excessive model-free play. In this paper, to alleviate the problem of hallucination in text summarization based on LLMs, we propose CL2Sum, a method that combines Contrastive Learning (CL) and ICL for summarization. After analysing the generated summaries of LLMs and summarising their hallucination types, we provided the models with accurate summaries and summaries containing hallucinations as ICL instances, either automatically or artificially. It aims to guide the model to make accurate predictions according to positive samples while also avoiding hallucinations similar to those in negative samples. Finally, a series of comparative experiments were conducted on summary datasets of different lengths and languages. The results show that CL2Sum effectively alleviates the hallucination problem of text summaries while also improving the overall quality of the generated summaries. Moreover, it can be widely adapted to text summarization tasks in different scenarios with a certain degree of robustness.