Retrievability in API-Based "Evaluation as a Service"

"Evaluation as a service" (EaaS) refers to a family of related evaluation methodologies that enables community-wide evaluations and the construction of test collections on documents that cannot be easily distributed. In the API-based approach, the basic idea is that evaluation organizers provide a service API through which the evaluation task can be completed, without providing access to the raw collection. One concern with this evaluation approach is that the API introduces biases and limits the diversity of techniques that can be brought to bear on the problem. In this paper, we tackle the question of API bias using the concept of retrievability. The raw data for our analyses come from a naturally-occurring experiment where we observed the same groups completing the same task with the API and also with access to the raw collection. We find that the retrievability bias of runs generated in both cases are comparable. Moreover, the fraction of relevant tweets retrieved through the API by the participating groups is at least as high as when they had access to the raw collection.