<p>Visual-based defect detection efficiently monitors the health and quality of construction and industrial products. However, current defect detection methods often improve detection accuracy at the cost of lower inference speeds or more parameters, struggle with complex data representation, emphasize target features while neglecting environmental information importance, and utilize convolutional or max pooling operations for downsampling, leading to more feature loss. To address these issues, this work presents a low complexity, accurate defect detection network augmented by environmental information-assisted and flexible activation functions to enhance the neural network performance on complex data representation. Environmental information-assisted module is designed for defect detection tasks to assist in accurately locating and predicting defects. Moreover, this work restructure features post-downsampling to mitigate feature loss and design a simple feature module called deep-global fusion that integrates deep and global features to enhance detection performance. Extensive experiments validate the superiority of the proposed detection network. The deployment of the network on edge computing devices confirms its competitive advantage in portability and reliability.</p>