Recognizing Primary User (PU) activities in cognitive radio networks is indispensable for dynamic spectrum utilization. Due to stringently limited spectrum observing time, the problems of data deficiency and category imbalance hamper traditional spectrum sensing methods from agilely discerning the PU activities with high accuracy. In this paper, we define multiple PU transmitters’ joint operating modes as the PU activity scenes and propose a local descriptors-aided adaptive prototype rectification network, named LDAPRNet, to pinpoint the scene type. Specifically, to cope with the data deficiency problem at first, we use a Convolutional Neural Network (CNN) to extract Local Descriptors (LDs) from the few spectrum samples available. Then, we construct a classifier that utilizes the extracted LDs to identify the scenes, where the imbalanced categories are given equal attention by introducing the prototypical network. Experimental results show that the LDAPRNet achieves significant accuracy and generalization ability improvements compared to traditional methods, and therefore, it can serve as an easy-to-implement candidate solution for few-sample-based spectrum sensing.