Electron beam welding (EBW) is the preferred technique for joining TC4 titanium alloy. Accurately characterizing the welding keyhole and residual stress distribution, along with analyzing their formation mechanisms, is of paramount importance for ensuring high reliability of the TC4 EBW structures. In this regard, a “thermal-fluid-metallurgical-mechanical” multi-field coupling numerical method was developed, which was then validated by the vital experiments. The evolution of the keyhole, molten pool temperature, phase volume fraction and residual stresses was then revealed. The influencing mechanism of keyhole on residual stress was explored comprehensively. The results show that metal vapor recoil pressure serves as the primary factor in keyhole formation, while the surface tension promotes keyhole closure. The β-phase in the weld zone undergoes a complete transformation into α′ acicular martensite. Moreover, the maximum longitudinal stress occurs at the weld center, while the transverse stress exhibits a substantial stress gradient along the thickness direction. Increasing the welding power raises the temperature of molten pool. The keyhole depth and width are also enlarged, accompanying by the increase of residual stress, which is expected to offer a theoretical foundation for managing the keyhole and residual stress generated during the EBW of TC4 titanium alloy.