Among studies conceptualizing quality of life (QoL) as the subjective measure of one’s well-being, evidence suggests a link between cognitive impairment and diminished quality of life. However, the direction of the association is not clear, and most studies have employed a global measure of cognition, which can mask subtle domain-specific declines. In this study, we aim at examining the longitudinal associations between memory and QoL in a representative sample of adults and older adults in Europe and Israel, employing data from the Survey of Health, Ageing and Retirement in Europe (SHARE). The sample was composed by 56,616 respondents aged 50 or older at the beginning of the study period. A Parallel-Process Latent Growth Model (PP LGM) of memory and QoL conditioned on the effects of age, gender, educational level, physical inactivity, depressive symptomatology and social activity participation, was tested. Results displayed a trajectory of decline for both memory and QoL, with higher initial levels of memory associated with higher initial levels of QoL, and steeper decline in memory associated with steeper decline in QoL. Moreover, a positive effect of initial memory onto QoL trend over time was found, indicating that better memory is associated to better QoL in the future, but the opposite did not occur. Results also provide evidence of gender differences. All in all, this work found evidence supporting the longitudinal effect of memory on QoL, which may have consequences for intervention implementation, given that actions for alleviating age-related memory impairment could also have a positive impact onto older adults’ QoL.