This paper investigates the development of robust Speaker-Independent Speech Emotion Recognition (SI-SER) systems, which are essential for enabling more empathetic and natural human-computer interaction. To address the challenges of real-world deployment, we present a comprehensive analysis of publicly available SER datasets for the construction of SI-SER models. We perform rigorous cross-dataset validation, revealing the difficulty in achieving generalization across unseen speakers and domains, as reflected in initially low accuracy scores. To overcome this, we explore the aggregation of datasets as a strategy to improve generalization. Our results show that the combination of datasets yields significant performance gains, with an average accuracy improvement of 14.98% for the six datasets considered and a maximum improvement of 24.45%. We conduct a comparative study on the individual contributions of each dataset, revealing their distinct influences. Our analysis demonstrates that aggregation significantly enhances performance, suggesting a shared feature space for SER across diverse datasets.