<p>Open set recognition has attracted increasing attention in recent years. Over-confident predictions on unknown samples make it difficult to differentiate between known and unknown samples. In this study, we reveal that the redundant representation is a crucial reason for the over-confidence issue and unveil the potential for improving open set recognition performance by eliminating redundancy. To learn compact representations, we propose a new concept, named Minimal Discriminative Learning (MDL), by drawing on the advantages of discriminative learning. Specifically, we propose a simple yet effective redundancy reduction module to directly learn the compact representation by regularizing features and classifier weights without losing discriminative ability. In addition, we propose to use the logits instead of the output of softmax as the open set scores, considering that softmax cancels out the magnitude information of feature representation. Extensive experiments are conducted on SVHN, CIFAR, and TinyImageNet, and results show that MDL obtains significant gains compared with baseline methods and outperforms state-of-the-art methods in terms of AUROC/F1 scores for open set recognition.</p>