Design and Implementation of Data Acquisition System Based on Scrapy Technology

Abstract

In this paper, a data acquisition system based on the Scrapy crawler framework was designed and implemented, which can not only obtain data according to the user’s own needs, but also manage its own collection tasks simply. The Django MTV mode is used for development, and the underlying data collection framework applies Scrapy, an asynchronous crawler application framework implemented by Python. The web page analysis uses the method in combination of XPath and regular expression. The jQuery tree plug-in zTree is utilized to realize tree management of tasks, the bootstrap to achieve the effect of task name with the keyword combination query and page.