Hierarchical Categorization of Big Content Using Concept Topology

Abstract Methods that are both computationally feasible and practically effective are needed to make sense of big corpuses of content, or “big content.” For example, supervised categorization techniques for open-access academic publishing are ill-suited for automated categorization because they rely on an existing categorization scheme, but no supervised scheme can stay abreast of the rapidly evolving landscape of scholarly work. This problem also applies to any domain with very large document corpuses where no good categorization scheme exists. To address this challenge, we present an unsupervised method to fit a hierarchical categorization scheme to a corpus based on clustering the network of shared concepts in the corpus, or its “concept topology.” Our method potentially applies to any type of content, and it scales to large networks of millions of vertices. We have demonstrated the application of our method to a corpus of 1.5 million scholarly texts representing the majority of open access (OA) academic publications on the web, validating our results using expert librarian annotations. We have made our datasets openly accessible for research by others. We believe that our resulting categorization scheme best represents OA academic publishing as it exists today.