Recently, a Multi-scale Representation Attention based deep multiple instance learning Network (MRAN) has been proposed to directly extract patch-level features from gigapixel whole slide images, and achieved promising performance on multiple popular datasets. However, it directly employs noisy bag-level labels for model training, thereby restricting the model performance. To overcome this issue, we propose a novel Knowledge Distillation based Dual-Branch deep multiple instance learning framework, namely KDDB. Specifically, it integrates knowledge distillation in MRAN to generate soft targets for bag-level images, and designs a novel attention-semantic consistent loss to reduce the semantic inconsistency between the attention weights of bag-level images and their prediction class probabilities, so as to reduce the negative effect caused by bag-level wrong labels. Extensive experiments on multiple whole slide image datasets demonstrate the superior performance of the proposed framework over MRAN and other recent state-of-the-art methods, with better model interpretability. All source codes will available online.