There is considerable research to demonstrate that caregiving has a negative effect on the wellbeing of caregivers. However, much of this evidence relies on cross-sectional data. The limited longitudinal research on carers' health has been mixed and more recently, the support for the physical health and mortality effects of caring has been questioned. Importantly, there is a scarcity of studies that have accounted for the impact on the carer of the "family effect" of caregiving. The family effect is caused by caring about an ill, disabled, or frail significant other, while the caregiving effect is a result of caring for that significant other. Despite the potential to contribute to the overestimation of the caregiving effect on health, partialling out the family effect from the impact of providing care is rare. Using data from a nationally representative longitudinal study of ageing, we compare the health quality of life of three groups: those caring for (1) a spouse, (2) a parent, or (3) a friend (family effect). There were no significant differences between the three groups on physical health. Spousal carers had poorer mental health than those caring for parents or friends even when controlling for hours of care provided, dementia diagnosis and living arrangements. Quality of the care relationship and socioeconomic status were also strong predictors of mental health. These associations remained at 2-year follow-up. These distinctions are important as they require different responses from a public health perspective e.g., counselling services versus in-home support. We discuss the potential for developing more precise, culturally appropriate measures of the "family effect" for future research.