A Crawling Method with No Parameters for Geo-social Data based on Road Maps

Abstract

Researchers must crawl geo-social data to analyze and visualize geo-social data. A conventional method to exhaustively crawl geosocial data is based on a grid. The crawler divides a specified area into a grid and uses the center coordinates of each cell to query databases using APIs. However, there is a difficult problem when using the grid-based method. It is that researchers cannot estimate the optimized grid size to exhaustively crawl geo-social data in advance because the optimized grid size depends on data density owing to geographical characteristics of an area. We focus on the fact that geo-social data are dense along roads. Thus, we propose a method based on road maps to exhaustively crawl geo-social data. We demonstrated that our method can crawl geo-social data by using almost the same number of queries compared to the crawler with an optimized grid size.