Mria: An Eventually Consistent Mnesia

Abstract

Mnesia, an Erlang distributed database, serves as an embedded storage and replication layer for OTP applications requiring low read latency and high availability. EMQX is a publish-subscribe message broker supporting the MQTT protocol that uses Mnesia to replicate its internal state across the cluster. We analyze the limitations of Mnesia's replication protocol scalability in large clusters under high load. To address these limitations, we developed Mria, an extension to the Mnesia database that provides eventual consistency within a cluster and achieves better horizontal scalability. We validated Mria using a variety of testing techniques, including model checking, chaos engineering, and formal verification. Replacing Mnesia with Mria allowed us to scale the EMQX cluster to 23 nodes, handle 100 million simultaneous client sessions and achieve a higher sustained load.