Infrared small target detection (ISTD) technology is pivotal in both military and civilian sectors. Currently, detection methods based on deep learning have achieved certain results. However, there is still a lack of comprehensive interaction between features at different scales, resulting in the loss of some effective information and limiting the performance of the model. Moreover, some existing methods do not pay attention to parameter optimization, and the long model inference time fails to meet the lightweight and real-time requirements of actual scenarios. Therefore, we propose the lightweight and multi-scale adaptive network (LMANet). Firstly, we design the Residual Spatial and Channel Attention Refinement Block (RAR) in the feature extraction stage and combine it with the full-scale skip connections to maintain deep target features while processing redundant features to reduce the number of model parameters. Secondly, we design the bi-branch upsampling module (BU) to avoid image distortion caused by the loss of important information during the upsampling process. Additionally, we propose a layered iterative form of multi-scale deep adaptive supervision module (MSDAS) to fuse and supervise features of different scales to varying degrees to obtain accurate segmentation boundaries and improve detection performance. The intersection over union (IoU) of LMANet on the NUAA-SIRST and IRSTD-1k datasets is 78.06% and 66.12% respectively. Experimental results show that our method outperforms other methods.