Pre-trained language models (PLMs) have proven effective for named entity recognition (NER) tasks. However, with hundreds of millions of parameters, PLMs are too resource-hungry and computation-intensive to deploy in real-world applications. To alleviate this problem, one solution is to utilize knowledge distillation (KD) with in-domain unlabeled data to compress PLMs. Despite its wide use, in-domain data are not always available while open-domain data are plentiful. Unfortunately, the effects of open-domain data are not always guaranteed, sometimes even deteriorating original models. The main reason is that the open-domain data contain task-irrelevant samples, so the data distribution differs from that of the task data, leading to biased models. To address this issue, we propose BalanceKD, a framework that filters samples based on entity-level weight and constructs a balanced batch to reduce the noise and bias introduced by open-domain data. Compared to previous works, BalanceKD employs several innovations to use open-domain data while keeping promising performance. Concretely, we first rank samples from the open domain based on the entity-level weight and then select the top-k samples for training. Subsequently, we sample an equal number of samples from the task data and the open-domain data. Specially, we find that the open-domain data might mislead models at the early training stages and propose a novel warmup to help models remain stable. Our experiments on a wide range of NER benchmarks demonstrate that the proposed framework boosts the compressed models’ performance.