Transfer of Reinforcement Learning-Based Powertrain Controllers From Model- to Hardware-in-the-Loop

Abstract

Developing powertrain control functions is time-consuming and resource-intensive, often leading to sub-optimal solutions. Reinforcement Learning (RL) allows agents to perform complex control tasks with minimal human involvement, but is often confined to simulations due to testing costs and safety concerns. To effectively apply RL in embedded powertrain control, agents must be able to handle real-world scenarios, particularly through direct interaction with real actuators and control systems. Therefore, this research applies Transfer Learning (TL) and X-in-the-Loop (XiL) simulations to develop agents that can seamlessly transition and perform robustly in real-world environments. For transient exhaust gas re-circulation control of an internal combustion engine, the process begins with a computationally inexpensive Model-in-the-Loop (MiL) simulation to select a suitable algorithm, fine-tune hyperparameters, and conduct preliminary training. In the next step, pre-trained agents are transferred to an advanced Hardware-in-the-Loop (HiL) system with real hardware using TL for further training. Compared to agents trained entirely on HiL systems, transferred agents required significantly less real-world training time (up to $5.9$ times shorter) while outperforming the series production Engine Control Unit (ECU). The results highlight that for real-world effectiveness, integrating actual hardware into training is essential, reward fine-tuning plays a critical role in optimizing these interactions, and the maturity of the policy significantly influences both training duration and overall performance.