It is usually assumed that performance on non-verbal intelligence tests reflects visual cognitive processing and that aspects of working memory (WM) will be involved. However, the unique contribution of memory to non-verbal scores is not clear, nor is the unique contribution of vocabulary. Thus, we aimed to investigate these contributions. Non-verbal test scores for 17 individuals with intellectual disability (ID) and 39 children with typical development (TD) of similar mental age were compared to determine the unique contribution of visual and verbal short-term memory (STM) and WM and the additional variance contributed by vocabulary scores. No significant group differences were found in the non-verbal test scores or receptive vocabulary scores, but there was a significant difference in expressive vocabulary. Regression analyses indicate that for the TD group STM and WM (both visual and verbal) contributed similar variance to the non-verbal scores. For the ID group, visual STM and verbal WM contributed most of the variance to the non-verbal test scores. The addition of vocabulary scores to the model contributed greater variance for both groups. More unique variance was contributed by vocabulary than memory for the TD group, whereas for the ID group memory contributed more than vocabulary. Visual and auditory memory and vocabulary contributed significantly to solving visual non-verbal problems for both the TD group and the ID group. However, for each group, there were different weightings of these variables. Our findings indicate that for individuals with TD, vocabulary is the major factor in solving non-verbal problems, not memory, whereas for adolescents with ID, visual STM, and verbal WM are more influential than vocabulary, suggesting different pathways to achieve solutions to non-verbal problems.