Clustered Federated Learning (CFL) has attracted extensive attention due to its ability to cluster clients into different clusters based on data distribution, thereby mitigating the impact of the inherent Non-IID characteristics of client data. Existing CFL methods often involve slow convergence and inflexible clustering strategies. To address these issues, this paper proposes FedPSAC, an innovative CFL algorithm that effectively identifies similarities between data distributions by analyzing the prototypes of client data and adaptively discovers the clustering structure of clients without the need to predefine the number of clusters through the Affinity Propagation algorithm. The server receives the prototypes extracted from local clients and uses a pre-trained neural network model to train the prototypes to enhance their representational capabilities. The server performs adaptive clustering by calculating the similarity of client prototypes and then aggregates models within each cluster. Additionally, a unique inter-cluster model optimization strategy is designed to enhance the convergence and generalization capabilities of cluster models. Extensive experiments on three datasets under three Non-IID settings demonstrate that FedPSAC outperforms several popular FL baseline methods in terms of average test accuracy in data heterogeneous scenarios.