On Combining Publish/Subscribe with Append-only Logs for IoT Data

Publish/subscribe systems and in particular the MQTT protocol are often used to cope with the messaging needs of modern Internet of Things (IoT) systems, i.e. between the IoT devices and the related services. Their benefits lie in their decoupling properties, easing the seemless movement of the systems components across the increasingly distributed hardware substrate of cloud, fog, and edge. However, publish/subscribe systems alone are rarely suitable for all usecases commonly associated with the IoT. One important example is the lack of persistent storage of sensor data in popular pub/sub system, such as MQTT, which often leads to the anti-pattern of constantly subscribing and locally storing the data at the subscriber side, resulting in multiple independent copies accross the network. In this paper, we present our analysis of integrating storage technologies into pub/sub: We show how append-only logs, considered to be the prevailing storage paradigm in some IoT focused systems, can be added to common pub/sub systems. Three options are outlined in detail regarding features. We further show on an abstract level how the glue code combining the solutions might look like.