The guided image filter is a widely used technique for smoothing images, and it can be applied to various low-level vision tasks. However, it suffers from issues such as luminance halos, detail halos, and corner round-up artifacts. Additionally, the original guided image filter often overlooks important thin structures due to the \(L_2\) regularization in the linear ridge regression model. Existing adaptations of the guided image filter do not adequately address these artifacts. In this paper, we propose a novel scale-aware guided image filter, which effectively smooths out details/textures while preserving edges/structures. To overcome the luminance halos, we propose to incorporate the elastic net regression model for image smoothing, which inherits the advantages of both ridge and lasso regression. Furthermore, we propose a scale-adaptive weighting scheme to combine the regression coefficients, which significantly alleviates the detail halo and corner round-up artifacts. Despite various improvements, we show that the model can be solved with high efficiency. We have conducted experiments across various low-level vision applications, including image smoothing, compression artifact removal, HDR tone mapping, and texture removal. Both qualitative and quantitative results indicate the superiority of our proposed filter over state-of-the-art filters. Moreover, our filter is efficient and allows for real-time smoothing of 720P 3-channel color images.