We used fMRI to identify brain areas that adapted to either animals or manipulable artifacts while participants classified highly-rendered color photographs into subcategories. Several key brain areas adapted more strongly to one class of objects compared to the other. Namely, we observed stronger adaptation for animals in the lingual gyrus bilaterally, which are known to analyze the color of objects, and in the right frontal operculum and in the anterior insular cortex bilaterally, which are known to process emotional content. In contrast, the left anterior intraparietal sulcus, which is important for configuring the hand to match the three-dimensional structure of objects during grasping, adapted more strongly to manipulable artifacts. Contrary to what a previous study has found using gray-scale photographs, we did not replicate categorical-specific adaptation in the lateral fusiform gyrus for animals and categorical-specific adaptation in the medial fusiform gyrus for manipulable artifacts. Both categories of objects adapted strongly in the fusiform gyrus without any clear preference in location along its medial-lateral axis. We think that this is because the fusiform gyrus has an important role to play in color processing and hence its responsiveness to color stimuli could be very different than its responsiveness to gray-scale photographs. Nevertheless, on the basis of what we found, we propose that the recognition and subsequent classification of animals may depend primarily on perceptual properties, such as their color, and on their emotional content whereas other factors, such as their function, may play a greater role for classifying manipulable artifacts.