Fluorescence microscopy enables high-resolution imaging and is widely used in biomedical research. Accurate image segmentation is essential for quantitative analysis, but diverse imaging conditions and targets hinder model generalization. While vision models such as DINO exhibit strong generalization on natural images, their performance on fluorescence microscopy is limited by domain gaps. Moreover, DINO-based self-supervised learning focuses on intra-image structures and is limited in learning cross-image semantic correspondence between foreground and background regions. In fluorescence microscopic images, however, the foreground is consistently highlighted by fluorescent labeling, while the background remains uniformly dark, making it well-suited for foreground-background contrastive learning. Therefore, we propose a segmentation framework that combines domain-specific self-supervised pretraining with cross-image foreground-background contrastive fine-tuning. Specifically, we pretrain a Vision Transformer using fluorescence microscopy images, and introduce cross-image foreground-background contrastive learning during fine-tuning to enhance the model’s ability to distinguish semantic boundaries and generalize across datasets. Experiments on multiple fluorescence microscopic image segmentation datasets show that our framework improves the average IoU and Dice scores by 6.00% and 6.40%, respectively, compared to models trained from scratch, and achieves improvements on unseen biomarkers. Our code and pretrain weights are available at https://github.com/MoriLabNU/FMI-ViT