We study the spread of susceptible-infected-recovered (SIR) infectious diseases where an individual's infectiousness and probability of recovery depend on his/her "age" of infection. We focus first on early outbreak stages when stochastic effects dominate and show that epidemics tend to happen faster than deterministic calculations predict. If an outbreak is sufficiently large, stochastic effects are negligible and we modify the standard ordinary differential equation (ODE) model to accommodate age-of-infection effects. We avoid the use of partial differential equations which typically appear in related models. We introduce a "memoryless" ODE system which approximates the true solutions. Finally, we analyze the transition from the stochastic to the deterministic phase.