Older male premutation carriers of the FMR1 gene are associated with the risk of developing a late-onset neurodegenerative disorder, fragile X-associated tremor/ataxia syndrome. Although previous postmortem and in vivo magnetic resonance imaging studies have indicated white matter pathology, the regional selectivity of abnormalities, as well as their relationship with molecular variables of the FMR1 gene, has not been investigated. In this study, we used diffusion tensor imaging to study male premutation carriers with and without fragile X-associated tremor/ataxia syndrome and healthy sex-matched controls. We performed a tract of interest analysis for fractional anisotropy and axial and radial diffusivities of major white matter tracts in the cerebellar-brain stem and limbic systems. Compared with healthy controls, patients with fragile X-associated tremor/ataxia syndrome showed significant reductions of fractional anisotropy in multiple white matter tracts, including the middle cerebellar peduncle, superior cerebellar peduncle, cerebral peduncle, and the fornix and stria terminalis. Significant reduction of fractional anisotropy in these tracts was confirmed by voxel-wise analysis using tract-based spatial statistics. Analysis of axial and radial diffusivities showed significant elevation of these measures in middle cerebellar peduncle, even among premutation carriers without fragile X-associated tremor/ataxia syndrome. Furthermore, regression analyses demonstrated a clear inverted U-shaped relationship between CGG-repeat size and axial and radial diffusivities in middle cerebellar peduncle. These results provide new evidence from diffusion tensor imaging for white matter abnormalities in the cerebellar-brain stem and limbic systems among individuals with the fragile X premutation and suggest the involvement of molecular mechanisms related to the FMR1 gene in their white matter pathology.