Deep neural networks have shown remarkable performance across diverse machine learning tasks. However, the balance between predictive accuracy and model interpretability remains a persistent challenge: high-performing models often exhibit complex structures defying human understanding, while interpretable (concise) models may sacrifice performance. In this paper, we show that feature interaction can be a crucial perspective when pursuing such balance, and propose flexible-order feature-interaction (FOFI), a new approach to exploit grouped feature interactions as the key to building accurate yet interpretable models. FOFI encourages local feature interactions that are organized into groups, which allows model capacity (parameters) to be distributed in a nuanced manner: at the lower granularity, dense interactions are restricted locally within each group to account for the complexity (performance); at the higher granularity, a flat predictive function is defined at group-level that guarantees the overall interpretability. Furthermore, FOFI is versatile in accommodating feature interactions of arbitrary order among mixed continuous and categorical variables. Extensive experiments on both simulated and real-world datasets showcase the encouraging performance and interpretability of FOFI.