Recently, foundation models have emerged as versatile tools in computer vision, exhibiting remarkable performance across various tasks. Among these, Segment Anything Model (SAM) has been introduced for segmentation tasks using prompt-based learning. This model is trained using large datasets, providing a generalized understanding of visual data in real-world imaging domains. Consequently, current research focuses on utilizing their potential for specific domains, such as in medical imaging. The lack of training data and concerns regarding patient privacy present significant challenges in acquiring large medical datasets for model training. Therefore, the utilization of foundation models holds significance in effectively addressing these challenges. To this end, in this paper, we propose a liver lesion detection framework that combines the foundation model-SAM with a feature extraction network and detection head. To effectively extract the relevant information, we employ a gate-based mechanism designed to leverage the knowledge extracted in the SAM model within the detection network for lesion detection from multiphase CT images. We also utilize an auxiliary segmentation branch to enhance the primary lesion detection task, by taking advantage of multitask learning. The Experiment results demonstrate that our proposed framework achieves competitive results compared to existing state-of-the-art methods.