Near-space hypersonic vehicles could experience both continuous and rarefied flow regimes during the flight through atmosphere. The rarefaction effects on hypersonic boundary-layer stability are studied based on the Navier-Stokes (NS) equations and an improved NS model. The conventional linear stability theory (LST) is extended for rarefied shear flows by adopting slip boundary conditions and nonlinear transport relations. A flow at Mach 10 over a flat plate at an altitude of 55 km is the main case of analysis. The separate and combined effects of rarefaction (including surface slip and shear nonequilibrium) on stability by influencing the base flow and stability equation are clarified. The results show that for the base flow, rarefaction effects cause the boundary layer to become thinner and the generalized inflection point to move towards the wall. For stability, rarefaction effects have a stabilizing effect on the second-mode instability by influencing the base flow while a destabilizing effect by modifying the stability equation. The combined effects of rarefaction suppress the second-mode instability for different Mach number cases. However, for the first-mode instability, rarefaction effects play a destabilizing role. These results shed light on the hypersonic boundary-layer stability in the near-continuum regime from a macroscopic view.