Web forum crawling using text-based filters

Abstract

Forum websites contain very valuable information about frequently occurred problems in any subject. The good thing about these forums is, it is organized in a very proper format so that people can navigate easily to their expected discussion page. Now a days search engines are becoming very specific for different types of websites. So, they are making special crawlers for different types of websites to crawl exact information from raw html code. The research on forum crawlers is increased since last few years. But the main challenge in forum crawlers is finding similarity between URLs from different forum sites. This paper focuses on crawling forum URLs and classifying them as Entry, Index, Thread and Page Flipping URLs.