Let A be a unital \(C^*\) -algebra and U(A) be the unitary group of A. For \(x\in A\) , the unitary orbit of x is the set \(\{u^*xu: u\in U(A)\}\) , we denote by \({\mathcal {U}}(x)\) the closure of the unitary orbit of x. In this paper, we show that if A is a unital simple \(C^*\) -algebra of stable rank one and real rank zero, and \(V_1,V_2\in U(A)\) with \([V_1]_1=[V_2]_1=0\) in \(K_1(A)\) , then \(D_c(V_1,V_2)=\textrm{dist}({\mathcal {U}}(V_1),{\mathcal {U}}(V_2))\) , where \(D_c(V_1,V_2)\) is a spectral distance function introduced by Hu and Lin and \(\textrm{dist}({\mathcal {U}}(V_1),{\mathcal {U}}(V_2))\) is the distance between \({\mathcal {U}}(V_1)\) and \({\mathcal {U}}(V_2)\) . Furthermore, we show that if A is a unital simple \(C^*\) -algebra of tracial rank zero, and \(V_1,V_2\in U(A)\) with \([\lambda -V_1]_1=[\lambda -V_2]_1\) for all \(\lambda \notin {\mathbb {T}}\) in \(K_1(A)\) , then \(D_c(V_1,V_2)=\textrm{dist}({\mathcal {U}}(V_1),{\mathcal {U}}(V_2))\) . Thus, we generalize the results by Bhatia and Davis for distance between unitary orbits of unitary matrices.