This study investigated the effects of 35 years of application of inorganic fertilizers containing nitrogen (N), phosphate (P), or potassium (K) alone or in combination on the abundance and composition of diazotrophic community in a black soil (Mollisol) in northeast China. The abundance and composition of diazotrophic community were analyzed using qPCR and Illumina MiSeq sequencing targeting nifH genes. Nitrogen fertilization decreased the abundance and Shannon diversity of nifH genes. The diazotrophic community was dominated by Alphaproteobacteria (Bradyrhizobium at the genus level), with relatively higher abundance in the N fertilization treatments than in the non-N fertilization treatments. All diazotrophic communities were clustered into two groups with and without N fertilization history, and the soil pH, total C, total N, and NO3--N significantly influenced the structure of the whole diazotrophic community. Moreover, random matrix theory analysis elucidated a clear difference in network structures between the N and non-N fertilization treatments, with N fertilization causing a less stable network structure. These results highlighted that it was N fertilizer, but not P and K fertilizers, contributed to great changes in the diazotrophic community in this black agricultural soil.