Alzheimer’s disease (AD) is the leading cause of dementia. Although there is currently no cure for AD, early detection of cognitive decline can help clinicians mitigate its impact. Recently, Machine Learning (ML) approaches have been developed to automatically analyze handwriting and hand-drawing tasks to support the early diagnosis of AD. In this paper, we study pentagon and clock drawing tests using both off-line (scanned image pixels) and on-line (discrete point sequences) data as input to several ML models (i.e., DensNet, ResNet, EfficientNet, RNN, LSTM, and GRU). Our study is the first to determine the most effective modality (on-line vs. off-line) and drawing tasks to distinguish healthy controls from AD patients (binary classification) as well as two stages of AD severity (multi-class classification). Our results suggest that, contrary to other domains, the off-line modality outperforms the on-line one, sometimes by a large margin: 90% vs. 60% accuracy in binary classification and 53% vs. 82% accuracy in multi-class classification. This suggests that, for drawing tasks and small-scale datasets, image-based representations may be more effective in predicting AD than those relying on more complex data representations.