Providing transactional quality of service in event stream processing middleware

We present an efficient algorithm for providing transactional guarantees when federating a transactional object store with a stream processing engine. The object store allows transactions to programmatically access and update persistent tables as inmemory map objects. The stream processing engine processes continuous queries on events from designated map objects to generate continuously updated derived views. Stream processing traditionally provides an “eventual correctness” guarantee; however our clients insisted upon a uniform transactional quality of service for the federated system: in particular, the system must guarantee that no transaction can observe an updated map object and a stale value of a derived view. A straightforward approach would entail executing all stream processing within the scope of transactions, and this comes at a performance cost. Our approach, based on a hybrid optimistic concurrency control technique, allows stream processing to proceed in parallel, usually after commit time. We present our algorithm along with experimental performance evaluations of both approaches.