This paper presents a genealogy of the passport in international law. For the most part, the origins of our contemporary mobility order are narrated from the vantage point of the grand principles of sovereignty, hospitality, and liberty. The nuts and bolts through which people access mobility – passports and visas – are generally understood to be the natural, inevitable, and fair by-products of these principles. This paper contributes to existing debates on the coloniality of international migration law by examining the universalization of the passport under the League of Nations. I argue that the universalization of the passport meant abandoning the old idea that the rights of free movement belonged to everyone and, instead, instituted a system that ranks human mobility based on national origins. Theoretically, it is proposed that attention to these “lowly” practices of mobility governance allows us to track the afterlives of race in the international order.