In the open set object detection(OSOD) task, the model needs to identify both the known categories seen during training and unknown categories that have not been seen. However, unknown objects are often mistakenly identified as known objects by object detectors. To solve this issue, we propose a feature space clustering optimization method based on Aggregate Prototype Convergence(APC) and Contrastive Prototype Separation(CPS). This method enhances the clustering effect of known categories through more discriminative category boundaries, and makes the unknown categories far away from the known objects in the feature space. Consequently, this approach can effectively detect unknown objects. Specifically, an Aggregate Prototype Convergence learning method is first used to cluster similar features near their respective prototypes, enhancing the compactness of intra-class features. Then, the Contrastive Prototype Separation method is designed to maximize the inter-class distance from prototype to prototype, which significantly improves the separation of inter-class features. Meanwhile, to ensure that the distribution of similar features is not too scattered or concentrated, Boundary Constraint Loss (BCL) is designed to further optimize the feature distribution by setting upper and lower bounds on the feature space of each class. In addition, we incorporate an object focus module to predict object scores and enhance the detection of unknown classes. Experimental results under various open-set conditions show that the proposed model improves the unknown object detection recall (Ru) by 1.23%–1.46% and decreases the absolute open-set error (AOSE) by 19.05%–33.79% compared to similar methods, demonstrating excellent performance in open-set scenarios.