Towards large-scale data discovery: position paper

Abstract

With thousands of data sources spread across multiple databases and data lakes, modern organizations face a data discovery challenge. Analysts spend more time finding relevant data to answer the questions at hand than analyzing it. In this paper we introduce a data discovery system that facilitates locating relevant data among thousands of data sources. We represent data sources succinctly through signatures, and then create search paths that permit quick execution of a set of data discovery primitives used for finding relevant data. We have built a prototype that is being used to solve data discovery challenges of two big organizations.