Amplifying commonsense knowledge via bi-directional relation integrated graph-based contrastive pre-training from large language models

Abstract

Commonsense knowledge graph acquisition (CKGA) is vital in numerous knowledge-intensive applications such as question-answering and knowledge reasoning. Conventional CKGA methods rely on node-level and unidirectional relations, making them suffer from a shallow grasp of between entities and relations. Moreover, they also demand expensive, labor-intensive human annotations, and the yielding CK lacks diversity and quality. Existing commonsense knowledge bases such as ConceptNet or ATOMIC often struggle with significant scarcity and pose a major challenge in meeting the high demand for a vast amount of commonsense information. Given the recent momentum of large language models (LLMs), there is growing interest in leveraging them to overcome the above challenges.

In this study, we propose a new paradigm to amplify commonsense knowledge via bi-directional relation integrated graph-based contrastive pre-training (BIRGHT) from the newest foundation models. BRIGHT is an integral and closed-loop framework composed of corpora construction, further contrastive pre-training, task-driven instruction tuning, filtering strategy, and an evaluation system. The key of BRIGHT is to leverage reverse relations to create a symmetric graph and transform the bi-directional relations into sentence-level ones. The reverse sentences are considered positive examples for forward sentences, and three types of negatives are introduced to ensure efficient contrastive learning, which mitigates the “reversal curse” issue as evidenced in experiments. Empirical results demonstrate that BRIGHT is able to generate novel knowledge (up to 397K) and that the GPT-4 acceptance rate is high quality, with up to 90.51% (ATOMIC) and 85.59% (ConceptNet) accuracy at top 1, which approaches human performance for these resources. Our BRIGHT is publicly available at https://github.com/GreyHuu/BRIGHT/tree/main.