Query routing: applying systems thinking to Internet search

Abstract

As the Internet continues to grow exponentially, locating desired information becomes more difficult. Centralized approaches to the problem may prove unable to cope with this growth, and existing distributed ones do not scale well either, so it would be wise to investigate alternatives. One alternative is 'query routing', a technology which uses compressed indices from multiple sites to 'route' queries to where they may be able to be fulfilled. By using a referral model for the clients, it should be possible to provide a service which scales for both clients and servers. Server scaling is achieved by index compression along several novel axes. Allowing the generation of false positives permits the use of techniques which have higher compression. We have been able to add techniques from distributed naming systems such as DNS to traditional indexing based on flat sets of attribute value pairs. We have implemented a prototype that incorporates several of these techniques.