Designing a Modular and Distributed Web Crawler Focused on Unstructured Cybersecurity Intelligence

There are many use cases for cybersecurity related information available on the Internet. Tasks relating to natural language processing and machine learning require large amounts of structured and labeled data. However, the availability of recent data is limited due to the difficulty in its sanitization, retrieval, and labeling. Data on the Internet is generally diverse and unstructured, and storing this information in a manner that is easily usable for research and development purposes is not an intuitive task. We propose architectural considerations when developing a distributed system consisting of web crawlers, web scrapers, and various post-processing components, as well as possible implementations of these considerations. Our team developed such a system that is capable of applying structure and storing open source intelligence data from the Internet in an easily-searchable software platform called Splunk.