Stress affects cardiovascular, immune, and psychological health, making early detection essential. This paper focuses on stress detection using blood volume pulse data from photoplethysmography (PPG) sensors, which enable low-cost, non-invasive, and continuous monitoring. Recent work applied deep learning, mainly convolutional neural networks (CNNs) with image encodings (e.g., scalograms, Gramian angular fields) of time series, outperforming traditional methods. We extend this direction by benchmarking a Vision Transformer (ViT), an underrepresented architecture in PPG-based stress detection, against CNNs and ConvNeXts. Across encodings, scalograms yielded the best results for all architectures. In three-class stress detection, ViT-B/32 achieved the highest accuracy with the lowest variance, followed closely by CNNs. In binary classification, performance differences diminished. The accuracy in the out-of-domain evaluation dropped across all models, but was higher on datasets with greater similarity in design, reflecting the difficulty of generalisation under differing experimental conditions.