A programming framework for integrating web-based spatiotemporal sensor data with MapReduce capabilities

Abstract

Web-based sensor data, provided by organizations such as the National Oceanographic and Atmospheric Administration, provide a valuable service to the public and scientific communities. However, much of this data is locked in a variety of presentation formats and is computationally inaccessible. In addition, although these data have a spatiotemporal context, both the spatial and temporal data are usually only implicitly defined. Although storing this data in a consistent database can partially resolve this problem, a data-driven programming model coupled with MapReduce capabilities is a more flexible and extensible solution. Our implementation of this programming model allows users to parse a wide array of sensor data and express complex computation in a simple, scalable manner. In addition, our framework uses a simple key-value storage mechanism and provides convenient geospatial output mechanisms. In this paper, we discuss some early results of our programming model within the context of our current Java-oriented implementation, and demonstrate how the system can be used to create many different applications. We also discuss and evaluate our system with respect to memory usage and scalability.