The relationship between β-amyloid deposition and memory deficits in early Alzheimer's disease is unresolved, as past studies show conflicting findings. The present study aims to determine the relative contribution of regional β-amyloid deposition, hippocampal atrophy and white matter integrity to episodic memory deficits in non-demented older individuals harbouring one of the characteristic hallmarks of Alzheimer's disease, i.e. with β-amyloid pathology. Understanding these relationships is critical for effective therapeutic development. Brain magnetic resonance imaging and [(11)C]Pittsburgh Compound B-positron emission tomography scans were obtained in 136 non-demented individuals aged over 60 years, including 93 healthy elderly and 43 patients with mild cognitive impairment. Voxel-based correlations were computed between a memory composite score and grey matter volume, white matter volume and β-amyloid deposition imaging datasets. Hierarchical linear regression analyses were then performed using values extracted in regions of most significant correlations to determine the relative contribution of each modality to memory deficits. All analyses were conducted pooling all groups together as well as within separate subgroups of cognitively normal elderly, patients with mild cognitive impairment and individuals with high versus low neocortical β-amyloid. Brain areas of highest correlation with episodic memory deficits were the hippocampi for grey matter volume, the perforant path for white matter volume and the temporal neocortex for β-amyloid deposition. When considering these three variables together, only hippocampal volume and temporal β-amyloid deposition provided independent contributions to memory deficits. In contrast to global β-amyloid deposition, temporal β-amyloid deposition was still related to memory independently from hippocampal atrophy within subgroups of cognitively normal elderly, patients with mild cognitive impairment or cases with high neocortical β-amyloid. In the pre-dementia stage of Alzheimer's disease, subtle episodic memory impairment is related to β-amyloid deposition, especially in the temporal neocortex, and independently from hippocampal atrophy, suggesting that both factors should be independently targeted in therapeutic trials aimed at reducing cognitive decline.