The microstructure and mechanical properties of Mg-8Gd-2Nd-xEr-0.5Zr alloy were systematically studied. The effect of Er addition on the microstructure of as-cast alloy was discussed by calculating the mismatch degree, and the strengthening mechanism of aged alloy was expounded from the aspects of fine grain strengthening and precipitation strengthening. The mismatches between (100), (110), and (111) planes of Mg5Gd and (0001) plane of α-Mg were 0.95, 2.33, and 1.65%, respectively. Calculations demonstrate that Mg5Gd can serve as an effective core for α-Mg nucleation. The addition of the Er element can replace Gd atom to form Mg5 (Gd, Er) phase, which increases the effective heterogeneous nucleation core, thus refining the grains. In addition, the number of β′ precipitates increases after aging treatment, which better hinders the dislocation movement and improves the performance. The ultimate tensile strength (UTS) of Mg-8Gd-2Nd-3Er-0.5Zr alloy can reach 313, 319, 307 and 287 MPa at 25, 200, 250 and 300 °C, respectively. The fracture morphology has a large number of dissociation surfaces and tearing edges, showing brittle fracture.