Large language models (LLMs) have shown remarkable capabilities in scientific domains, spurring industry interest in their use for aiding academic paper comprehension. One of the critical components of this task is understanding the mathematical formulas embedded within scholarly texts, which are often highly symbolic and abstract, encapsulating the core logic and reasoning of the research. However, the proficiency of LLMs, as well as the applications that integrate them, in comprehending these mathematical expressions remains unclear. To address this, we introduce FORCE, a novel benchmark comprising 15,106 formula-centered questions derived from 4,074 academic papers. FORCE is built using an automated pipeline that extracts formulas and context to construct a knowledge graph (KG), refined via self-correction before generating diverse reasoning-based question-answer pairs. Our evaluation on FORCE uncovers a significant performance disparity between surface-level textual comprehension and deeper formulaic reasoning. This finding underscores a critical area for the industry to address when advancing applications that integrate LLMs for academic paper understanding.