Vision transformers are essential for medical image analysis thanks to their ability to capture long-range dependencies. However, their quadratic computational cost with sequence length challenges high-resolution 3D tasks such as diffusion models or inpainting. FlashAttention eases memory bottlenecks via local access patterns, yet the computational load remains high. Quantising or binarising weights and activations shows promise in CNNs but often degrades accuracy and requires highprecision training. In transformers, work has focused on quantised linear layers or sparse attention, while binary attention remains largely unexplored. Originally presented as oral at MIDL 2025 [1], our work introduces a novel differentiable binary attention mechanism that enables 1-bit precision computation of self-attention during both training and inference. Our method combines bitwise hamming distances with learnable scalar weighting of queries and keys to provide gradients. In theory it achieves 16–32× improvements in computational and memory efficiency over floating-point attention. We evaluate BinaryFormer on challenging tasks with sequence lengths of N>1000: image classification without patch embedding, semantic 2D MRI segmentation, and 3D high-resolution diffusion modelling for inpainting and synthesis. Across all tasks, our binary attention achieves competitive performance while greatly reducing resource demands. For faster inference, binarisation-aware training with a straight-through estimator is sufficient, whereas our hamming Attention is essential for fully binary precision training. Future work will explore integrating binarisation into other transformer components, e.g. linear weight matrices, and investigating alternative backpropagation schemes using trainable attention biases. These directions may further enhance the efficiency of low-precision transformers for medical imaging and beyond. Code is available at https://github.com/mattiaspaul/binaryformer.