As the visual system ages, flicker sensitivity decreases and the latencies of cortical visual evoked potentials (VEP) increase. However, the extent to which these effects reflect age-related changes in the magnocellular (M) and or parvocellular (P) pathways remain unclear. Here, we investigated the relation between flicker fusion frequencies and VEP non-linearities induced by rapid stimulation, as a function of age over 6 decades. The approach, using Wiener kernel analysis of multifocal flash (mf)VEP, allows the extraction of signatures of both M and P processing and hence establishing a neural basis of the known decline in flicker fusion threshold. We predicted that, in a sample of 86 participants, age would be associated with a latency increase in early mfVEP response components and that flicker fusion thresholds, for both low and high contrast stimuli, would relate to the temporal efficiency of the M-generated VEP component amplitudes. As expected, flicker fusion frequency reduced with age, while latencies of early second order peaks of the mfVEP increased with age, but M temporal efficiency (amplitude ratio of first to second order peaks) was not strongly age-related. The steepest increases in latency were associated with the M dominated K2.1 (second order first slice) N70 components recorded at low and high contrast (6.7 and 5.9 ms/decade, respectively). Interestingly, significant age-related latency shifts were not observed in the first order responses. Significant decreases in amplitude were found in multiple first and second order components up to 30 years of age, after which they remained relatively constant. Thus, aging and decline in visual function appears to be most closely related to the response latencies of non-linearities generated by the M pathway.