An Efficient Approach for Determining Optimal Number of Clusters
摘要
The elbow method, based on the Sum of Squared Errors (SSE), is a widely used approach for determining the optimal number of clusters in the K-Means clustering algorithm. This method involves a linear search across varying values of the number of clusters \( k \) (typically ranging from 2 to \(\sqrt{n}\) , where \( n \) is the number of data points), followed by the evaluation of a cluster validity index such as the Silhouette Score. However, this approach is computationally expensive, as it requires checking each value of \( k \) sequentially. In this paper, we propose an alternative approach that employs a binary search mechanism on the Silhouette Scores computed for \( n \) clusters to efficiently identify the optimal cluster count that maximizes the Silhouette Score. To mitigate the risk of local optima, we further validate the selected cluster count using the Davies-Bouldin (DB) index. The effectiveness of the proposed method is demonstrated on multiple real-world datasets, establishing its superiority over the conventional approach.