Images of buildings and manipulable objects have been found to activate distinct regions in the ventral visual pathway. Yet, many non-categorical properties distinguish buildings from common everyday objects, and perhaps the most salient of these is size. In this fMRI study, we investigated whether or not changes in perceived scale can account for some of the differences in category-specific responses, independent of the influence of semantic or retinotopic image properties. We used independent scans to localize object-specific ROIs in lateral occipital cortex (LO) and scene-specific ROIs in the parahippocampal place area (PPA) and posterior collateral sulcus. We then contrasted the effects of stimulus category and perceived size/distance in these regions in a factorial design. Participants performed an oddball detection task while viewing images of objects, buildings, and planar rectangles both with and without a background that indicated stimulus size/distance via simple pictorial cues. The analyses of fMRI responses showed effects of perceived size/distance in addition to effects of category in LO and the PPA. Interestingly, when simple rectangles were presented in a control condition against the background that indicated size/distance, LO in the right hemisphere responded significantly more to the small/close rectangles than to the large/far ones, in spite of the fact that the rectangles themselves were identical. These findings suggest that ventral stream regions that show category specificity are modulated by the perceived size and distance of visual stimuli.