Abnormal eye movements can potentially serve as symptoms of Parkinson’s disease (PD). With the advancements in artificial intelligence technology, an increasing number of studies have focused on the diagnosis and staging of PD through the analysis of eye movements. However, existing hand-crafted features-based machine learning methods have limitations in extracting the latent features of PD eye movements. To address these issues, we design a smooth pursuit eye movement task to collect eye movement data from 41 PD patients and 47 healthy controls, and propose a deep learning framework for the diagnosis of PD patients based on eye movement features. This work uses Bidirectional Long Short-Term Memory network, Visual Geometry Group network, Residual Network, and InceptionTime network to extract deep features from the eye movement sequence data. The effectiveness of these four methods is validated on the collected dataset. The results show that the InceptionTime network demonstrates superior capability in extracting eye movement features. Furthermore, pupil diameter is most influential in making accurate PD diagnosis.