Face identity aftereffects suggest that an average face, which is continuously updated by experience, functions as a norm for coding identity. Sex-contingent figural face aftereffects indicate that different norms are maintained for male and female faces but do not directly implicate them in coding identity. Here, we investigated whether sex-specific norms are used to code the identities of male and female faces or whether a generic, androgynous norm is used for all faces. We measured identity aftereffects for adapt-test pairs that were opposite relative to a sex-specific average and pairs that were opposite relative to an androgynous average. Identity aftereffects are generally larger for adapt-test pairs that lie opposite an average face, which functions as a norm for coding identity, than those that do not. Therefore, we reasoned that whichever average gives the larger aftereffect would be closer to the true psychological norm. Aftereffects were substantially and significantly larger for pairs that lie opposite a sex-specific than an androgynous average. This difference remained significant after correcting for differences in test trajectory length. These results indicate that, despite the common structure shared by all faces, identity is coded using sex-specific norms. We suggest that the use of category-specific norms may increase coding efficiency and help us discriminate thousands of faces despite their similarity as patterns.