This study focuses on the channel selection problem for femto base stations (FBSs) that share channels with macro base stations (MBSs). We formulate the femtocell scheduling problem within the framework of restless multi-armed bandits (RMAB). Our objective is to select channels that maximize the expected discounted return over an infinite horizon while minimizing interference to the macrocell caused by the shared channels with femtocells. Since the true channel state is not directly observable, we utilize the available feedback from users known as the channel quality indicator (CQI). Generally, the RMAB problem is recognized as PSPACE-hard. To tackle this challenge, we derive a closed-form approximation for the channel index by employing an iterative method. Based on this approximated index, we improve an existing policy called the approximated Whittle index policy, which provides a low-complexity solution for ranking the available channels for FBSs. Furthermore, we demonstrate the superior performance of the proposed algorithm compared to the existing approach.