We present HistoryCLIP, an adaptive multi-modal retrieval system, to address challenges of imbalanced and long-tailed historical archives with a comprehensive study on the utility of adapting vision-language models to such systems characterized by long-tailed sample distribution. We investigate three distinct fine-tuning techniques: full parameter fine-tuning, LoRA, and progressive layer unfreezing and benchmarked our results against the baseline in a diverse collection of historical archival data while created via stratified random sampling. Our extensive experiments with quantitative evaluation and qualitative analysis on both image-to-text and text-to-image retrieval tasks demonstrate that tailored fine-tuning substantially improves retrieval performance, offering critical insights into the adaptation of language-vision models for complex, domain-specific tasks. The proposed framework not only advances retrieval quality in the context of historical archives but also contributes to the broader understanding of fine-tuning methods for imbalanced multi-modal datasets.