Open source intelligence (OSINT) for conflict monitoring in contemporary South Africa: Challenges and opportunities in a big data context

Abstract

ABSTRACT With the advent of the information age, Open Source Intelligence (OSINT) has gained special prominence in the Intelligence Community (IC). However, the era of big data has brought numerous challenges in handling OSINT, in particular because big data comprise large volumes of unstructured data that are generated continuously. In this article, we discuss the use of OSINT in the era of big data, in particular how it relates to compiling event data, but also show that large projects with an international focus such as the Integrated Crisis Early Warning System (ICEWS) are inadequate to study political instability in the current South African context. We shift our collection efforts from mainstream media to the analysis of WhatsApp messages, since this platform has recently gained popularity and it is widely used in South Africa. We show how we build an automated data pipeline that provides near real time data on the occurrence of mass violence in South Africa. For this analysis, we make use of Natural Language Processing (NLP) toolkits in Python and build interactive dashboards to monitor the state of mass violence in South Africa.