Optimization-based spectral end-to-end deep reinforcement learning for equity portfolio management

Abstract

We propose a novel approach to equity portfolio optimization that combines spectral analysis and classical equity portfolio optimization theory with deep reinforcement learning in an end-to-end framework. We introduce the End-to-end Frequency Online Deep Deterministic Policy Gradient (EFO-DDPG) algorithm, which leverages discrete Fourier transform to decompose asset return sequences into frequency components. Unlike traditional methods that treat high-frequency components as noise, EFO-DDPG learns to adjust the influence of different frequency components dynamically. Moreover, the algorithm embeds a mean–variance portfolio optimization problem within a deep learning network, enhancing interpretability compared to black-box approaches. The framework models the investment problem as a Partially Observable Markov Decision Process (POMDP), using a state processing block with transformer encoders to capture complex relationships in the market data. By integrating spectral analysis, portfolio optimization theory, and online deep reinforcement learning, EFO-DDPG aims to adapt to non-stationary financial markets and generate superior investment strategies.