Self-interference (SI) suppression has always been critical for simultaneous transmit and receive (STAR) systems. To address the strong coupling between nearby transmit and receive antennas, this study employs an improved quantum genetic algorithm (IQGA) to eliminate SI by combining digital self-interference cancellation (SIC) and adaptive beamforming (ABF). Through digital SIC, both transmit noise and transmit signals are suppressed. The objective of the transceiver beamformer is to further reduce SI through SIC while simultaneously achieving high transceiver gain in the desired direction. IQGA represents the weights of feasible transceiver beams using chromosomes. By updating the population through quantum rotation, quantum crossover and quantum mutation strategies, IQGA demonstrates improvements in convergence accuracy, convergence speed, and reliability. In comparison to traditional ABF approaches, IQGA eliminates the need for complex matrix calculations and numerical derivations, thereby simplifying the solution process. The simulation results indicate that by cancelling the SI component through SIC and ABF, an isolation of 159.78 dB can be achieved at a transmit power of 1000 W. This represents an improvement of 40.32 dB compared to SIC alone and 113.87 dB compared to scenarios without SIC and ABF.