Rotor stability analysis is essential to ensure rotating composite structures' safe and efficient operation. In this paper, the stability analysis of a hybrid composite shaft with two disks placed on elastic supports is investigated by the three-node finite element method. The strain potential energy of the hybrid composite shaft is calculated by considering the Timoshenko beam theory and using shape functions and the ABD matrix's effective components. The governing equations of the composite rotor are derived by replacing the kinetic energy of the shaft and disks, the strain potential energy, and the force of the bearings in the Lagrange equation. The equations of motion obtained from the finite element method are coded in the state space using MATLAB script. Their eigenvalues are calculated as a function of the rotor rotation speed, and the instability threshold of the composite rotor is evaluated. To validate the simulation results of the composite shaft in the free–free state in the ANSYS software, a hybrid composite shaft is made using the filament winding method, and its natural frequencies are extracted by performing the experimental modal analysis test. The instability threshold of the non-hybrid composite rotor of the presented model in different stacking sequences is compared with the results of the previous studies, and the validity of the three-node finite element method is confirmed. Finally, the effect of the fiber angle and the arrangement of the layers in the usage of carbon/epoxy and glass/epoxy in a specific stacking sequence on the stability of the hybrid composite rotor is studied.