Towards a Benchmark for Fact Checking with Knowledge Bases

Abstract

Fact checking is the task of determining if a given claim holds. Several algorithms have been developed to check facts with reference information in the form of knowledge bases. While individual algorithms have been experimentally evaluated, we provide a first publicly available benchmark evaluating fact checking implementations across a range of assumptions about the properties of the facts and the reference data. We used our benchmark to compare algorithms designed on different principles and assumptions, as well as algorithms that can solve similar tasks developed in closely related communities. Our evaluation provided us with a number of new insights concerning the factors that impact the performance of the different methods.