Trust is crucial for effective human-robot interaction in assistive robotics. This paper presents a proof-of-concept study for transformer-based trust prediction, building on our previous work in understanding and measuring trust. We introduce a novel multi-modal video transformer to predict user trust in human-robot interactions with an autonomous wheelchair. We collected a comprehensive dataset comprising over 850,000 tokens, equivalent to approximately 4 h of egocentric video streams, along with synchronized continuous trust recordings, eye gaze patterns, physiological data, and trust measurements from 34 diverse participants. Our research utilizes Trusty, a tool for continuous trust measurement, which has been previously validated against traditional methods. Our model utilizes a pre-trained Vision Transformer (ViT) with frozen weights to extract spatial features from video frames. Visual representations are enriched with the user’s eye gaze, heart rate, and electrodermal activity in a fusion module. Fusion outputs are temporally processed by a transformer, and the final token in the sequence is used to predict trust. This architectural design leverages transformers’ proven capabilities in processing high-dimensional visual data, effectively fusing signals across modalities, and scaling to larger datasets. The best-performing model achieves approximately 64% accuracy in predicting trust levels (high, medium, low), with eye gaze data significantly enhancing performance. Future work will focus on personalisation and real-time adaptive robot behaviour based on trust predictions.