Explaining the predictions of neural models in clinical NLP remains a significant challenge, especially for complex tasks involving long, unstructured medical texts. While post-hoc methods like LIME and SHAP are widely used, they often fall short when applied to clinical narratives. In this paper, we identify core limitations of token-level and perturbation-based explanation techniques through targeted demonstrations on a hospital length-of-stay prediction task. Our findings reveal issues such as overemphasis on non-informative tokens, instability in attributions, and high-confidence predictions for incoherent input variants. These results underscore the need for explanation strategies that are clinically meaningful, semantically grounded, and robust to linguistic noise.