Plasmonic gold nanorod instability and reshaping behavior below melting points are important for many future applications but are yet to be fully understood, with existing nanoparticle melting theories unable to explain the observations. Here, we have systematically studied the photothermal reshaping behavior of gold nanorods irradiated with femtosecond laser pulses to report that the instability is driven by curvature-induced surface diffusion rather than a threshold melting process, and that the stability dramatically decreases with increasing aspect ratio. We successfully utilized the surface diffusion model to explain the observations and found that the activation energy for surface diffusion was dependent on the aspect ratio of the rods, from 0.6 eV for aspect ratio of 5 to 1.5 eV for aspect ratio less than 3. This result indicates that the surface atoms are much easier to diffuse around in larger aspect ratio rods than in shorter rods and can induce reshaping at any given temperature. Current plasmonics and nanorod applications with the sharp geometric features used for greater field enhancement will therefore need to consider surface diffusion driven shape change even at low temperatures.