In Medical Visual Question Answering (Med-VQA), accurate interpretation of clinical questions alongside medical images is crucial for reliable diagnostic support. However, conventional methods often exhibit pronounced medical language biases that stem from imbalanced data distribution and question shortcut dependence, causing models to disproportionately rely on textual priors at the expense of valuable visual semantics. To mitigate this challenge, we propose a novel Med-VQA debiasing approach called “Med-BiasX” that synergistically combines two strategies, i.e., Energy-aware Confidence Constraint (ECC) and Distribution-aware Dependence Calibration (DDC). Specifically, ECC aims to reinforce correct answers and adjust the energy associated with incorrect answers by leveraging the global normalization property of free energy and the intrinsic properties of energy. DDC is designed to shift the model’s dependency from question shortcuts to multimodal information by explicitly measuring the similarity between predicted distributions from different branches and prior distributions. Extensive experiments on multiple medical standard benchmarks and bias-sensitive benchmarks, SLAKE-BIAS and VQA-RAD-BIAS, consistently demonstrate the robustness and superiority of our Med-BiasX approach over state-of-the-art competitors.