A Vision Transformer with Adaptive Cross-Image and Cross-Resolution Attention
摘要
Vision Transformers (ViTs) are the current state-of-the-art in deep learning for computer vision tasks. They are trained on vast datasets and are capable of useful downstream tasks through clever use of the attention mechanism. The biggest limiting factor for ViTs is the number of pixels and tokens that can be processed in a given pass. Memory constraints on both patch size and the number of patches mean that ViTs are most effective at processing relatively low-resolution images. Whilst ViTs can attend very flexibly across an image, attending across images in a naive fashion requires memory proportional to the square of the number of images. This is a further limiting factor. Given the task of automated assessment of psoriasis severity, a chronic skin condition that can affect large portions of a person’s skin, it is necessary to look across multiple images and at fine detail in large images. We present a method that adapts ViTs to a two-stage design that allows for the regression of a patient’s psoriasis score across multiple images and resolutions and shows its effectiveness relative to a baseline ViT. The implementation of our method is available at https://github.com/KCL-BMEIS/multivit.git .