The Segment Anything Model (SAM), introduced in 2023, has made significant advancements in the field of computer vision. However, SAM faces two major challenges: limitations related to single-scale processing and high computational demands for large-scale data handling, which restrict its widespread adoption in practical applications. Therefore, we propose MS-SAM to achieve better segmentation performance at a lower cost. We introduce a multi-scale image encoder to handle images of varying scales and complexities. Additionally, we propose a dynamic weighted agent attention, which can flexibly and efficiently adjust attention distribution while reducing computational costs. We train MS-SAM on a 1/20 subset of the SA-1B dataset for instance segmentation using points or boxes as prompts. The results show that we maintain the accuracy with the SAM method while reducing the parameters by about 6 times. We give sufficient experimental results to demonstrate its effectiveness.