Depth estimation techniques typically involve extracting features of objects in the environment and their relationships. However, these methods require multiple images, making them less feasible for real-time scenarios. To alleviate these challenges, the rise of efficient convolutional neural networks (CNNs) with the ability to infer depth from a single image opens a new avenue for investigation. Current research introduces an efficient FDE-Net designed to generate cost-effective depth maps from a single image. The new framework consists of a PPLC-Net as the convolutional encoder and a fast decoder as the decoder. Moreover, this combination integrates the Squeeze-Exploit (SE) module using the MKLDNN optimizer to enhance convolutional efficiency and rationalize model size with efficient training. Meanwhile, the proposed multi-scale pixel-wise fast decoder generates state-of-the-art depth maps while maintaining an efficient structure. Experimental results demonstrate that our model achieves state-of-the-art performance on four datasets: NYU-V2, KITTI, Cityscapes, and a simulated environment. Unexpectedly, FDE-Net utilizes merely 0.04 times the parameter counts of Resnet-Upconv. Computational efficiency is profoundly underscored by FLOP and MAC, showcasing a considerable superiority relative to competing models. FDE-Net exhibits a remarkably reduced latency of 4.2 times, in addition to a 3.9 times enhancement in throughput when contrasted with Resnet18-Upconv.