House rules are essential for P2P accommodation hosts to regulate guest behavior and manage their properties; however, they can also represent a double-edged sword for guests. This research proposes to examine the joint effects of house rules from the perspective of human territoriality, and highlights two psychological mechanisms—uncertainty reduction and psychological reactance—that underline the nonlinear relationship between the number of house rules and rental performance. A sample of 12,108 Airbnb listings was processed using semi-supervised machine learning techniques as well as ordinary least squares and Bayesian estimations. Findings revealed an inverted U-shaped relationship: the number of house rules initially had a positive impact on rental performance, but this effect turned negative beyond a certain threshold. This relationship was mitigated by customer review volume. Conditions for the existence of an inverted U-shaped relationship were formally tested via Bayesian posterior distributions. Experimental evidence provided further support for the proposed mechanisms.