A Machine Learning Approach for Disease Surveillance and Visualization using Twitter Data

Abstract

Insights from real-time disease surveillance systems are very useful for the public to take preventive measures against the diseases and it also benefits the pharmaceutical manufacturers in improving the sales of medicines for the particular disease and ensuring adequate availability of medicines when they are needed.A disease outbreak is an event wherein there is a rise in the number of positive cases for a disease in a short span of time. An outbreak can be limited to a particular region or time of the year. Diseases can be detected by several approaches, social media being preferred method due to availability of real-time data. Hence, data from social media, especially Twitter can be used to detect live events and monitor them efficiently. In order to detect diseases precisely, this paper proposes an approach wherein tweets, which are collected and pre-processed, can be effectively vectorized and clustered into the appropriate diseases with the use Agglomerative Clustering technique. The tweets can also be visualized using their geo information in order to generate zones which have high density of diseases. Such a surveillance system can be of use for early prediction of disease outbreaks, in turn facilitating faster and better handling of the situation.