Introducing the Cosine Clustering Index (CCI): A Balanced Approach to Evaluating Deep Clustering

Amidst the surge of Big Data, deep clustering emerges as a pivotal technique in machine learning, necessitating robust and interpretable evaluation metrics that align with its complexities. Traditional metrics, largely dependent on Euclidean distance, falter in capturing the essence of deep clustering in high-dimensional data. This paper proposes the Cosine Clustering Index (CCI), an innovative metric that leverages cosine distance to accurately assess clustering outcomes in deep learning contexts. Distinguished by its ability to evaluate intra-cluster cohesion and inter-cluster separation, CCI addresses the intricacies of deep clustering more effectively than existing metrics. Validation on benchmark datasets like IRIS, MNIST, and CIFAR10 showcases CCI’s enhanced interpretability, scalability, and its superior adaptability to the depth and dimensionality of data. The introduction of CCI marks a significant stride towards advancing clustering evaluation, promising a more profound understanding and analysis of deep clustering mechanisms.