Practitioners typically evaluate binary price direction forecasts using standard classification metrics (AUC, accuracy, log loss, F1 score). However, evidence from various domains suggests that these metrics often fail to reflect actual economic payoffs. This paper explores the relationship between classification performance and real-world financial utility in machine learning-based binary options trading. We systematically evaluate multiple learning algorithms using both classification and utility-based performance measures (e.g., Sharpe ratio) for trading binary options. Empirical results show that improvements in classification metrics do not necessarily lead to better trading outcomes. Instead, a utility-based evaluation is critical for estimating trading performance reliably. These results underscore the importance of adopting decision-centric evaluation frameworks that prioritize economic decision quality over statistical performance metrics in financial forecasting applications, such as binary options trading.