Ontology Property-based Adaptive Crawler for Linked Data(OPAC)

Abstract

Linked Data is expected to play an important role for interoperability at the data level for the future internet. One of the main applications of Linked Data might be semantic query processing. The current method of warehousing based query processing requires the crawling of all data periodically and also frequent crawling of data from distributed datasets of Linked Data to make the data as up-to-date as possible. Frequent crawling is required for dynamic data to meet the high freshness requirement of real time applications. Crawling large datasets may cause serious scalability problems. In this paper, we propose an Ontology Property-based Adaptive Crawler to alleviate this problem. Linked data are crawled adaptively based on the Change Frequency of the Documents and the Properties. Performance evaluation shows that this system can reduce overhead costs by more than 70% while maintaining a high freshness of data.