This study examined the role of leisure-time physical activity in reducing the impact of high life stress and time pressure on depression, a buffer effect, for mothers of infants. A direct association between leisure-time physical activity and depression, regardless of both sources of stress, was also tested. A sample of approximately 5,000 mothers of infant children completed questionnaires that measured demographic characteristics, frequency of participation in leisure-time physical activity, life stress, time pressure, and depression (depressive symptoms). Hierarchical multiple regression incorporating an interaction component to represent the buffering effect was used to analyze the data. Frequency of leisure-time physical activity was significantly associated with lower levels of depressive symptoms for both types of stress and acted as a buffer of the association between life stress and depressive symptoms, but did not buffer the influence of time pressure on depressive symptoms. These findings indicated that leisure-time physical activity assists in maintaining the mental health of mothers of infants; however, caution is needed when promoting physical activity for mothers who feel under time pressure.