High-risk plaque (HRP) detected by coronary CT angiography (CTA) is associated with increased risks of major adverse cardiovascular events such as heart attack. Current identification of HRP characteristics involves labor-intensive segmentation of plaques, requiring substantial time and expert knowledge. In this work, we propose a novel coronary cross-sectional Vision Transformer (ViT) framework that bypasses the need for explicit segmentation by directly predicting the presence of HRP. Our approach extracts cross-sectional slices along the coronary centerline, ensuring that the model focuses on the artery. By leveraging the standard patch-based input of ViT, we capture not only the coronary cross-section itself but also surrounding contextual information (e.g., adipose tissue). Furthermore, we incorporate multiple levels of detail by combining the cross-sections from proximal and distal positions with their corresponding CTA axial planes, forming a comprehensive cross-sectional representation. We also embedded the actual 3D position of each cross-section into the positional encoding of the Transformer to enhance spatial awareness. Experimental results of 3,068 coronary arteries demonstrate that our method outperforms conventional approaches, highlighting its potential to optimize clinical decision-making in the care of coronary artery diseases. The code is available at https://github.com/JZCambridge/ViTAL-CT-MICCAI25 .