MinCache: A hybrid cache system for efficient chatbots with hierarchical embedding matching and LLM

Abstract

Large Language Models (LLMs) have emerged as powerful tools for various natural language processing tasks such as multi-agent chatbots, but their computational complexity and resource requirements pose significant challenges for real-time chatbot applications. Caching strategies can alleviate these challenges by reducing redundant computations and improving response times. In this paper, we propose MinCache, a novel hybrid caching system tailored for LLM applications. Our system employs a hierarchical cache strategy for string retrieval, performing exact match lookups first, followed by resemblance matching, and finally resorting to semantic matching to deliver the most relevant information. MinCache combines the strengths of Least Recently Used (LRU) cache and string fingerprints caching techniques, leveraging MinHash algorithm for fast the resemblance matching. Additionally, Mincache leverage a sentence-transformer for estimating semantics of input prompts. By integrating these approaches, MinCache delivers high cache hit rates, faster response delivery, and improved scalability for LLM applications across diverse domains. Our experiments demonstrate a significant acceleration of LLM applications by up to 4.5X against GPTCache as well as improvements in accurate cache hit rate. We also discuss the scalability of our proposed approach across medical domain chat services.