Infrared small target detection has significant application value in military, security, medical and other fields. However, due to the phenomenon of fuzzy edge contours of infrared small targets during imaging, coupled with interference from complex background noise, the detection accuracy of existing algorithms is affected. In this paper, an infrared small target detection model with edge refinement and joint attention enhancement is proposed to address the above issues. To improve the localizing accuracy of small targets with fuzzy contours, an edge refinement module (ERM) is designed to enhance the edge features of targets by integrating edge information extracted via edge detection filters and finite difference methods. To establish internal dependencies of feature maps, a joint attention enhancement module (JAEM) is further designed by integrating self-attention mechanism and input-dependent depthwise convolution operation. In this way, the interference from complex background noise on small target detection could be suppressed. Moreover, a multi-scale feature fusion module (MFFM) is proposed to fuse enhanced feature at different scales effectively. Hence, the feature description ability of the model for small targets could be improved. Experimental results on two widely used infrared small target detection datasets demonstrate that the proposed model could effectively improve the detection accuracy of infrared small targets.