Binary neural networks (BNNs) have gained considerable attention due to their low power consumption. However, there is still a performance gap between BNNs and their full-precision counterparts. Previous researches focus on reducing quantization errors and optimizing approximate gradients to enhance BNN performance. In this paper, we attribute the degradation of BNN performance to the destruction of deep-layer features. The limited representation capability in deep layers leads to insufficient semantic information for accurate target recognition. To address this problem, we construct a multi-branch training framework, which can guide BNN training with unimpaired deep feature supervision. Specifically, a full-precision adapter is utilized in the deep layer to refine the degraded feature representation in binarized layers. The full-precision adapter has three advantages: 1) it introduces more accurate gradient information flow to stabilize training in the shared fully binarized layers; 2) it incorporates lossless semantic information of deep layers that guides the training of binary branch; 3) it is generic and compatible to diverse BNN structures. We conduct extensive experiments on the CIFAR-100 and ImageNet datasets to evaluate the proposed method. The comprehensive experimental results demonstrate that the proposed method can significantly improve the performance of existing BNNs without incurring additional computational overhead during inference.