Image-text retrieval methods based on transformers have achieved performance progress and found widespread applications in various fields. However, in streaming data scenarios, retrieval models tend to forget previously learned knowledge. Although some cross-modal continual learning methods focus on this issue, it is noted that the vanilla transformer encoder is not suitable for streaming data. Also, existing methods concentrate only on retaining coarse-grained representations, neglecting fine-grained information. In this paper, a continual learning method is proposed to mitigate the issue of catastrophic forgetting that plagues image-text retrieval. Our method consists of three components: a novel transformer encoder incorporating Augmented Shortcuts and Inter-task Attention (ASIA), a module of Focused Retention for Fine-grained Semantics (FRFS), and a module of Collaborative Constraint on Dual-modal Drift (C \(^{2}\) D \(^{2}\) ). ASIA employs Augmented Shortcuts to enhance the representation capacity of fine-grained features and utilizes Inter-task Attention to slow down the drift of important attention between previous tasks and the current task. Moreover, the FRFS preserves crucial fine-grained information from the previous model, and the C \(^{2}\) D \(^{2}\) maintains the relative position among representations aggregated according to fine-grained importance weights. To demonstrate the effect, we develop two benchmarks by modifying MS-COCO and Flickr30k for the continual learning setting. Experimental results on the two benchmarks show that our method outperforms existing continual learning methods in terms of preventing forgetting as well as balancing new and old knowledge.