Interactions between the visual system and the motor system during action observation are important for functions such as imitation and action understanding. Here, we asked whether such processes might be influenced by the cognitive context in which actions are performed. We recorded ERPs in a delayed go/no-go task known to induce bidirectional interference between the motor system and the visual system (visuomotor interference). Static images of hand gestures were presented as go stimuli after participants had planned either a matching (congruent) or nonmatching (incongruent) action. Participants performed the identical task in two different cognitive contexts: In one, they focused on the visual image of the hand gesture shown as the go stimulus (image context), whereas in the other, they focused on the hand gesture they performed (action context). We analyzed the N170 elicited by the go stimulus to test the influence of action plans on action observation (motor-to-visual priming). We also analyzed movement-related activity following the go stimulus to examine the influence of action observation on action planning (visual-to-motor priming). Strikingly, the context manipulation reversed the direction of the priming effects: We found stronger motor-to-visual priming in the action context compared with the image context and stronger visual-to-motor priming in the image context compared with the action context. Taken together, our findings indicate that neural interactions between motor and visual processes for executed and observed actions can change depending on task demands and are sensitive to top-down control according to the context.