Color difference evaluation is an important topic in colorimetry, and it has led to many methods of calculations over the years, which still aim to improve accuracy. This study aims to explore the efficacy of existing color difference measures in evaluating perceived color differences under varying contextual conditions. A novel experiment was conducted wherein participants evaluated color pairs against different backgrounds. This contribution presents the preliminary results of this study which compare the perceived color on varying backgrounds with some of the most used color differences. Overall, the results show that the precision gained by improving point-wise color metrics is pointless due to the high variability of color perception in varying contexts caused by spatial mechanisms of the human visual system. In conclusion, this study highlights the importance of creating alternative methods to measure color differences that reflect how we see colors in real life. Such a challenge could improve color research and its applications across different fields.