As robots increasingly enter human-centered environments, they must not only be able to navigate safely around humans, but also adhere to complex social norms. Humans often rely on non-verbal communication through gestures and facial expressions, and respect body motion and language, when navigating around other people, especially in densely occupied spaces. Consequently, robots also need to be able to interpret body motion as part of solving social navigation tasks. To this end, we present Gesture2Path, a novel social navigation approach that combines image-based imitation learning with model-predictive control. We observe the human and their environment with a neural network operating on streams of images, generating point-to-point navigation tasks solved with state-of-the-art model predictive control. We deploy our human-aware policy on real robots and showcase the effectiveness of our approach for the four gestures-navigation scenarios: left/right, follow me, and make a circle. We validated our method based on in-situ ratings of participants interacting with the robots. Our experiments show our method can interpret complex human gestures and use them to generate socially compliant trajectories for navigation tasks.