An approach for accessing data from hidden web using intelligent agent technology

Abstract

There is large amount of information available on web, which is hidden from users. This is because such information is not able to be accessed or indexed by traditional search engines. These search engines are only able to crawl information by following hypertext links. The forms which require login or any authorization process can be ignored by them. Hidden web refers to that deepest part of the Web which is not available for traditional Web crawlers. Obtaining the content from Hidden web is a challenging task. Today many web sites are containing pages that are dynamic in nature. This dynamic nature of web pages creates a problem for retrieving information for traditional web crawlers. The effort done to solve the given problem is discussed in brief. Then, a comparative study among the earlier defined architecture, considering various parameters, is also shown. By analyzing above methods a framework is proposed which uses an intelligent agent technology for accessing the hidden web.