First break picking is basic for seismic data processing, but traditional automatic picking methods face big challenges in the noise and irregular conditions, leading to poor performance in those complex scenarios. Deep learning models have made significant improvements on the picking accuracy, but their reliance on large-scale, high-quality training datasets restricts their practical application. To address these challenges, we introduce FB-SAM, a novel learning framework based on Meta’s foundation model Segment Anything (SAM). Specifically, our FB-SAM framework introduces an adapter-based parameter-efficient fine-tuning method, adapts an enhanced decoder to improve the picking performance over the original SAM decoder, and utilizes high-frequency features extracted from two-dimensional seismic data to boost picking accuracy with limited data. Experimental results on the real seismic data of first break picking demonstrate that FB-SAM not only surpasses typical semantic segmentation models but also outperforms the original SAM using conventional fine-tuning approaches.