In the size-weight illusion (SWI), small objects feel heavier than larger objects of the same mass. This effect is typically thought to be a consequence of the lifter's expectation that the large object will outweigh the small object, because objects of the same type typically get heavier as they get larger. Here, we show that this perceptual effect can occur across object category, where there are no strong expectations about the correspondence between size and mass. One group of participants lifted same-colored large and small cubes with the same mass as one another, while another group lifted differently-colored large and small cubes with the same mass as one another. The group who lifted the same-colored cubes experienced a robust SWI and initially lifted the large object with more force than the small object. By contrast, the group who lifted the different-colored objects did so with equal initial forces on the first trial, but experienced just as strong an illusion as those who lifted the same-colored objects. These results demonstrate that color cues can selectively influence the application of fingertip force rates while not impacting at all upon the lifter's perception of object weight, highlighting a stark dissociation in how prior information affects perception and action.