Crack detection is a key link to ensure the safety of building roads and the life of structures, and plays a crucial role in early warning and maintenance decision-making. However, cracks are often distributed in complex backgrounds as low contrast and irregular multi-scale. The down sampling-up sampling process is easy to cause information loss and spatial misalignment, resulting in disconnected artifacts of long cracks. The imbalance of categories makes it difficult for small cracks to get enough attention, which seriously weakens the recall rate and robustness of the model. Therefore, a crack detection model based on multi-scale dilated convolution and edge optimization (MDENet) is proposed. First, the multi-scale dilated Edge Module (MDEM) was used to parallel the multi-scale dilated convolution and edge algorithm, and the crack edge was refined to realize the complementary capture of global semantic information and local contour. Then, the deep features were divided into tokens by Tokenized KAN Phrase (Tok-KAN) and non-linear re-validation was performed to significantly improve the sensitivity to subtle cracks and suppress complex background noise. Finally, the residual channel attention module RCAM dynamically adjusted the channel weights in the skip connection layer, which kept key information and suppressed redundant interference. Experimental results show that MDENet achieves F1 scores of 87.43%, 74.29% and 73.09% on three public datasets, respectively, which are significantly better than mainstream models.