Semi-Supervised Federated Learning (SSFL) under the label-at-server (LAS) paradigm assumes that the server owns a small labeled set while numerous clients contribute only unlabeled data. However, current SSFL methods hinge on a single global supervision source, generating pseudo-labels from a global perspective locally and then fine-tuning on the scarce server labels, which collapses inter-client diversity and leads to pathological overfitting to the labeled subset. To tackle this dual imbalance, we exploit the latent information in local low-confidence samples rather than discarding them. Concretely, we present CAPE-SSFL, whose Candidate-Consensus Learning (CCL) assigns entropy-weighted multi-candidate pseudo-labels to low-confidence samples, whose Candidate-aware Prototype Contrastive Learning (CaPCL) pulls these samples toward global class prototypes and pushes them away from non-candidate semantics in representation space, and whose Entropy–Prototype Adaptive Aggregation (EPAA) dynamically adjusts client weights according to average uncertainty and prototype consistency, balancing exploration and convergence. Extensive experiments on three benchmark datasets under diverse non-IID settings demonstrate that CAPE-SSFL outperforms state-of-the-art methods while effectively enhancing inter-client diversity.