Feature-based knowledge distillation has been recognized a remarkably effective way to transfer informative knowledge from a complicated teacher model to a simple student model. However, for most knowledge distillation methods, the teacher model merely regards the feature knowledge as a supervisory information but neglects its guidance to the student model, leading to a large gap between the feature knowledge of the teacher and that of the student. To overcome this weakness, we propose a novel preliminary knowledge guided distillation that incorporates the layer-level features from the teacher as prior knowledge to guide the student to generate the guided features. The guided features can narrow the difference between the teacher knowledge and the student knowledge. Furthermore, to enhance the quality of teacher features, a Multi-Level Feature Fusion module is employed to integrate the rich context of the teacher features across different levels, which benefits to more comprehensive exploration of teacher features. We validate the superiority of our approach by performing experiments on three different tasks, i.e., Image Classification on CIFAR-100 and Tiny ImageNet datasets, Object Detection and Instance Segmentation on MS-COCO dataset, respectively, indicating more competitive performance than other typical approaches.