Under the double pressure of the onset of aging and declining fertility rate, improving the efficiency of spatial allocation of the rural labor force is of great significance in promoting the sustainable development of agriculture. The loess hilly region is a typical ecologically fragile area in China. This paper takes 422 administrative villages in Qin’an County of China’s loess hilly region as the evaluation unit. It combines spatial concentration, spatial interaction, and spatial mismatch index to reveal the characteristics of the relationship between rural population and agricultural production space from a multi-gradient perspective. The results show that: (1) Rural population and agricultural production space show a reverse unbalanced “center-edge” structure, with rural population clustering in the urban area and agricultural production space spreading to the periphery, with a significant unbalanced distribution. (2) Rural population and agricultural production space show diversified relationship characteristics, with significant inconsistency in their spatial concentration, overall low spatial interaction, and obvious spatial mismatch index, which greatly affects the rational utilization of human resources and land resources. (3) The characteristics of the relationship between rural population and agricultural production space are influenced by natural endowment, economic development level, and location conditions, showing a multi-gradient differentiation pattern. Based on the research results, this paper proposes a reasonable path for the coordinated development of rural population and agricultural production space in China’s loess hilly region from a multi-gradient perspective.