The association of physical frailty and psycho-social health is well established. However, it appears no study on the interaction of these conditions in determining the Health-related Quality of Life (HRQoL) has yet been attempted in India. Hence, present study aims to investigate the interaction effects of physical frailty and psycho-social health in determining the HRQoL. A cross-sectional survey was conducted on 500 community-dwelling rural older adults from West Bengal, India. The modified Fried’s Frailty instrument was assessed physical frailty status. Psycho-social characteristics investigated were depression, anxiety and stress using the 21-item DASS instrument, loneliness using the UCLA Loneliness instrument, and cognitive function using the Mini-Mental State Examination instrument. HRQoL was assessed by the 36-items Short Form health survey questionnaire. Overall prevalence of frail, pre-frail and non-frail was 26.2%, 60.2%, and 13.6%, respectively. One-way ANOVA highlighted that mean scores of psycho-social traits are significantly differed among different physical frailty status groups (p < 0.01). All the participants were clustered with their psycho-social characteristics instrument scores as clustering variables using Hierarchical Cluster Analysis to understand psycho-social health of the participants. Two-way ANCOVA showed significant interaction of physical frailty status (frail, pre-frail and non-frail) and empirically occurring clusters of psycho-social health (low, moderate and good) in determining the overall HRQoL [F(8,491) = 8.904,p < 0.01]. Thus, the study findings suggest that consideration of physical frailty and psycho-social health status will be a prudent approach for a better diagnosis of HRQoL and challenges related to the same.