Using open-source tools for web crawling and indexing of open content

Abstract

The Internet has made possible the access to thousands of freely available music tracks under the Creative Commons or Public Domain licenses. This number keeps growing on a yearly basis. In practical terms, it is extremely difficult to browse this huge music collection, because it is widely dispersed throughout multiple websites. The work presented on this paper addresses the problem of indexing this large collection of free music. This is a very relevant problem because currently there are no integrated database or index holding information about this music material. Indexing this content will allow, for instance, the development of music recommendation systems that will also work with noncommercial content. In this paper the authors present a system proposal that has been developed to tackle the available free music indexing problem and how this system can be integrated with other systems (such as music recommendation systems) to allow the end users to enjoy free and open music content.