Size constancy is the result of cognitive scaling operations that enable us to perceive an object as having the same size when presented at different viewing distances. In this article, we review the literature on size and distance perception to form an overarching synthesis of how the brain might combine retinal images and distance cues of retinal and extra-retinal origin to produce a perceptual visual experience of a world where objects have a constant size. A convergence of evidence from visual psychophysics, neurophysiology, neuropsychology, electrophysiology and neuroimaging highlight the primary visual cortex (V1) as an important node in mediating size-distance scaling. It is now evident that this brain area is involved in the integration of multiple signals for the purposes of size perception and does much more than fulfil the role of an entry position in a series of hierarchical cortical events. We also discuss how information from other sensory modalities can also contribute to size-distance scaling and shape our perceptual visual experience.