Can We Detect Bias in Political Fact-Checking? Evidence from a Spanish Case Study

ABSTRACTPolitical fact-checkers evaluate the truthfulness of politicians’ claims. This paper contributes to an emerging scholarly debate on whether fact-checkers treat political parties differently in a systematic manner depending on their ideology (bias). We first examine the available approaches to analyze bias and then present a new approach in two steps. First, we propose a logistic regression model to analyze the outcomes of fact-checks and calculate how likely each political party will obtain a truth score. We test our model with a sample of fact-checks from Newtral, a prominent Spanish fact-checker. Our model would signal bias under two assumptions: (a) all political parties are on average equally accurate in their statements; (b) the verification method gives precise instructions and is implemented systematically. We investigate this second assumption with a series of interviews with Newtral fact-checkers. We show that standard verification protocols are loosely implemented and therefore fact-checks reflect a set of journalistic decisions, rather than a bias in the statistical sense. We call for a more rigorous definition of verification methods as a pre-requisite for an unbiased assessment of politician’s claims.KEYWORDS: Fact-checkingpolitical partiesbiasnoiseimpartialitypublic opinion Disclosure StatementNo potential conflict of interest was reported by the author(s).Notes1 Politifact received a Pullitzer prize in 2009 – https://www.pulitzer.org/winners/staff-69 (accessed: 21st March 2022).2 We use verification as a synonym to fact-checking.3 The IFCN mentions replicability: “Signatories want their readers to be able to verify findings themselves. Signatories provide all sources in enough detail that readers can replicate their work, except in cases where a source’s personal security could be compromised.” – https://ifcncodeofprinciples.poynter.org/know-more/the-commitments-of-the-code-of-principles (accessed: 21st March 2022).4 We use this term loosely, mostly as a synonym to fact-checking, as in Brashier et al. (Citation2021).5 As in: Graves Citation2018.6 There is no evidence that politicians in mainstream parties that represent mainstream ideologies are consistently more or less accurate in their statements. Therefore, we will assume that they have equal probability of being more or less accurate.7 Description gathered from Newtral: https://www.Newtral.es/quienes-somos/ (accessed: 21st March 2022).8 https://ifcncodeofprinciples.poynter.org (accessed: 21st March 2022).9 https://www.Newtral.es/metodologia-transparencia/ (accessed: 21st March 2022).10 Not all the fact-checks in the sample are independently published nor have a separate web link: Newtral also publishes compilations of fact-checks in a single piece. As a result, we counted as one fact-check each time Newtral issued a verdict according to their scale, regardless the fact-check was published isolated or in a compilation. However, we ruled out those fact-checks that were duplicate and those that targeted foreign politicians.11 https://ifcncodeofprinciples.poynter.org/application/public/newtral/08442D12-61C6-3050-33F7-C0F003C26588 (accessed: 21st March 2022).12 The party membership of the claim’s author is adjudicated according to his/her then political appointment at the time the claim was made (e.g., member of parliament). Short of any appointment, the minimal requirement for affiliation was to be publicly known as member of the party. The affiliation was coded into a multinomial variable with unordered levels. The numeric code for each party is: 1 – PSOE (social democratic), 2 – PP (conservative), 3 – Ciudadanos (liberal), 4 – Podemos (far-left), 5 – Vox (alt right), 6 – Other (an array of mainly regional parties with diverse ideologies).13 1 – “True”, 2 – “Half true”, 3 – “Misleading” and 4 – “False”.14 There are relevant differences in authorship: Some of these fact-checkers signed just one piece in the year.15 There were 24 fact-checks signed by “Newtral” without any reference to a particular author. If we assume that the interviewed fact-checkers were responsible for the same proportion (69’8%) of the 24 unsigned fact-checks, then we would see that the fact-checkers we interviewed are responsible for 75% of the verifications in the sample.16 This study obtained approval by the Ethics Committee at Universidad Nacional de Educación a Distancia (UNED) – Reference code: 1-2021 FSOF.17 The arbitrary selection of an alternative baselines would not have changed any results.18 Ideologies are based on these parties’ group affiliation in the European Parliament.19 Model 2.20 https://www.newtral.es/metodologia-transparencia/ (accessed: 21st March 2022).21 “We choose all those statements that have interest or relevance from a purely journalistic criterion. We consider the relevance of the statement and the author, if it is repeated as an argument created intentionally to confuse and if it has verifiable content with data” (ibid.).22 One of the main goals stated in the IFCN Code of Principles: “Signatories want their readers to be able to verify findings themselves. Signatories provide all sources in enough detail that readers can replicate their work, except in cases where a source’s personal security could be compromised.” – https://ifcncodeofprinciples.poynter.org/know-more/the-commitments-of-the-code-of-principles (accessed: : 21st March 2022).Additional informationFundingThis work was supported by Ministerio de Ciencia e Innovación. Authors 1,2 and 4 received funding from the project: RTI2018-097709-B-I00.