Developing wind–photovoltaic–storage hybrid power system (WPS-HPS) is crucial for many countries seeking to advance their energy transition. However, the intricacies of both internal production processes and external market conditions often leads to suboptimal operational decisions, resulting in missed profit opportunities and abandoned renewable energies. To address this problem, this study builds an operational decision model aimed at maximizing profits while minimizing the renewable energy curtailment rate. Different from most studies that only consider one single market, the model constructed in this study comprehensively involves day-ahead–auxiliary service (DAAS) joint market, and is more suitable for the actual situations. As this model is essentially a complex nondeterministic polynomial problem, it cannot be solved directly through conventional programming methods. Consequently, this study proposes a hybrid metaheuristic non-dominated sorting dung beetle optimizer (HMNSDBO) to solve the above operational decision problem. Based on Dung Beetle Optimizer (DBO), the proposed HMNSDBO algorithm combines 6 improvement strategies. This effectively alleviates the problem of low initial population quality, premature convergence, and the tendency to fall into local optimum. Compared with DBO and other 4 popular algorithms, the search capability, optimization accuracy, and convergence speed of HMNSDBO are improved by 7.88 %, 16.1 %, and 6.95 %, respectively. When participating in DAAS joint market, compared with participating solely in day-ahead market or auxiliary service market, the proposed HMNSDBO enable WPS-HPS to achieve a profit increase of 21.68 % or 107.40 %, respectively, and a renewable energy curtailment rate decrease by 4.11 % or 73.98 %, respectively.