Toppling in layered anti-inclined rock slopes generally trends towards self-stabilization over long periods of time. This progressive process poses a distinct challenge in the accurate evaluation of the stability condition of an anti-inclined rock slope. The universal distinct element code (UDEC) was used in this study to reproduce the mechanics of a toppling failure. We developed two FISH functions to capture detailed joint damage and track the evolution of the interlayer normal forces. The initiating and resisting mechanisms of toppling were investigated based on the results of the numerical simulation. An improved limit equilibrium method, which considers the effects of the interlayer forces, was established to quantitatively evaluate the stability of anti-inclined rock slopes subjected to initial rotation. We proposed a ratio between the actual slope deformation and the deformation in the layer symmetry condition to determine the self-stabilization of toppling deformations. The results demonstrate that initial rotation of rock columns is prone to occur on steep slopes with larger layer dip angles and lower internal friction angles of the rock layers. Significant differentials between principal stresses create larger interlayer shear stresses, thereby facilitating the initial rotation of rock columns. The toppling deformation is considered to have reached a stable state when the sum of the orientations between the initial and deflected layers in the middle of the slope equals 180°. The case study indicates that the formation of gaps between layers and the downward movement of interlayer normal forces significantly reduce the slope stability, with \(\:{F}_{\mathrm{s}}\) decreasing from 1.99 to 1.01. As toppling deformations reach self-stabilization, the deformation factor of safety in the middle of the slope remains approximately 1.