Cooperative Non-Orthogonal Broadcast and Unicast Transmission for Integrated Satellite–Terrestrial Network

The integrated satellite-terrestrial network (ISTN) is gaining traction for providing seamless communication and various services, i.e., broadcast and unicast information services. However, meeting massive terminal access and diverse information services poses challenges due to limited spectrum resources and complex multiple access interference in ISTN. Recently, rate-splitting multiple access (RSMA) has emerged as a promising solution offering non-orthogonal transmission and robust interference management. Inspired by this, we design the non-orthogonal broadcast and unicast (NOBU) transmission model by utilizing the common and private data streams of RSMA. Taking different levels of cooperation between satellite and base station (BS) into consideration, we propose two cooperative NOBU transmission schemes, where one is that only broadcast messages are shared, and the other is that the broadcast message and the sub-common message split by terminals are shared and jointly encoded into a super-common stream. Building upon this, we formulate joint max-min rate optimization problems while satisfying the broadcast information rate requirement in ISTN. To address these non-convex problems, we introduce an improved alternating optimization algorithm based on weighted minimum mean square error. Simulation results validate the significant gains of cooperative NOBU schemes compared to various baseline schemes.