Efficient Compression and Preprocessing for Facilitating Large Scale Spatiotemporal Data Mining - A Case Study based on Automatic Identification System Data

Abstract

The large scale deployment of sensor, Global Positioning System (GPS) and other mobile devices generates large volumes of spatiotemporal data, which facilitates the understandings of objects' movement trajectories and activities. However, it is very challenging to store, transfer and load such a large volume of data into system memory for processing and analysis. In this study, we look into a study case that processes the large scale of Automatic Identification System (AIS) data in the maritime sector, and propose a computational framework to efficiently compress, transfer and acquire necessary information for further data analysis. The framework is composed of two parts: The first is a lossless compression algorithm that compresses the AIS data into binary form for efficient storage, speedy loading and easy transfer across networks and systems within the organization; the second is an aggregation algorithm which derives movement and activity information of vessels grouped by grid and/or time window from the compressed binary files, therefore improves data accessibility and reduces storage demand. The proposed framework has been applied to extract vessel movement information within Singapore port with high compression rate and fast access speed, and it can be extensively applied for other data processing applications.