PARL: Page Allocation in hybrid main memory using Reinforcement Learning

Abstract

Hybrid Main Memory introduces emerging non-volatile memory technologies and reduces the DRAM footprint to address the increasing capacity demands of modern workloads and DRAM scaling issues. The resulting heterogeneity requires new policies to distribute data and to minimize the potential workload slowdown. Existing literature proposes fixed-logic or machine learning-based solutions using naive and suboptimal initial data placement and focuses on the subsequent data migration policies. We explore this gap in the initial placement and propose to improve it using an adaptive and technology-agnostic solution. Page Allocation using Reinforcement Learning (PARL) is an agent that learns the target memory device for the initial placement of memory pages based on the rewards received from the system. PARL makes decisions by observing the state of its environment using system-level attributes instead of analyzing memory access patterns. This allows for a smaller state space and makes a reinforcement learning agent feasible for implementation, contrary to the claims found in the literature. Compared to fixed-logic methods, our proposal achieves 16%–43% better workload runtime and 21% better DRAM hitrate on average across the evaluated workloads. PARL also improves the DRAM hitrate by 9% on average (up to 34%), compared to a proposal using machine learning.