Keep a Human in the Machine and Other Lessons Learned from Deploying and Maintaining Colandr

Abstract

The rise of the “invest in open” movement has led to increased focus on - and recognition of the positive benefit of - open source toolkits as a way of democratizing access to software that is typically expensive and therefore restricted to research institutions. From 2015-2017, DataKind partnered with researchers from NCEAS through the SNAPP Consortium to tackle a problem common across many sectors: how to digest the amount of evidence available for decision-making in a way that would still allow for timely decisions to be made. Realizing that the evidence was stored primarily in PDFs and that the rise of machine learning techniques such as natural language processing meant that thousands of words from PDFs could be processed on local computers, the research team took an approach of building an open source tool to compete with commercially available evidence synthesis tools. With an algorithmic backend that relies on word vectorization, this project is an example of technology use to aid common labor intensive researcher tasks. From inception to early maintenance, this project produced many valuable lessons regarding the launch and stewardship of a public good and this article is a reflection of the learnings across that process.