In game theory, the emergence and maintenance of cooperative behavior within a group is a significant topic in evolutionary game theory and complex network theory. However, the limitations of a single mechanism in traditional networks restrict a thorough analysis of the sustenance and development of cooperative behavior, given the challenges posed by the diversity of social groups. To address this issue, this paper combines reinforcement learning game strategies with traditional prisoner's dilemma strategies based on two-layer coupled network to investigate the transmission of cooperative behavior among individuals in games. In our research, we study the evolutionary pattern and phase transitions using the Monte Carlo method. We use the prisoner's dilemma game as a mathematical model, establishing two subpopulations in each layer, with mutually payoff-neutral players between different subpopulations. This configuration results in intriguing spatiotemporal dynamics and patterns, leading to the spontaneous emergence of a cyclic dominance, where defectors from one group become prey for cooperators in another group, and vice versa. By simulating game evolution, we explore individual strategy changes and the impact of individual abilities on cooperative behavior in reinforcement learning. Extensive validations indicate that, in social dilemmas, adjusting the abilities of groups through effective guidance can sustain cooperative behavior. This guidance enables us to comprehend the stability of cooperation under adverse conditions. Simultaneously, the coexistence of two subpopulations greatly amplifies the complexity of evolutionary dynamics, causing a increase in cooperation rate.