This paper introduces ScanDDM, a novel scanpath model enabling generalised zero-shot goal-directed attention prediction. Leveraging recent advancements in vision-and-language learning techniques, ScanDDM models goal-directed attention by integrating high-level abstract concepts provided through textual prompts. The approach relies on a multialternative Drift Diffusion Model (DDM), framing gaze dynamics as a decision-making process that encapsulates both fixation duration and saccade execution. This allows to implement a value-based evidence accumulation process akin to the neurobiological mechanisms surmised to underlie human perceptual decision making. ScanDDM’s efficacy is quantitatively evaluated against the state-of-the-art model on the COCO-Search18 dataset, demonstrating excellent capabilities in predicting task-driven scanpaths in a zero-shot setting. Moreover, qualitative results showcase ScanDDM’s ability to generalize to complex and abstract concepts, beyond simple visual search tasks. Source code available at: https://github.com/phuselab/scanDDM .