The use of Large Language Models (LLMs) in information retrieval can lead to inaccurate, inconsistent, incomplete, irrelevant, or biased outputs. To address these risks, we argue that critical thinking serves as a powerful antidote, equipping users with the skills to navigate and mitigate these risks effectively. This paper examines leading conceptualizations of critical thinking and contextualizes its application in LLM-based information retrieval. We review state-of-the-art approaches for minimizing these risks and highlight their limitations. Building on this, we propose five novel Critical Thinking Support Functions, aimed at fostering user criticality during LLM interactions. We report on workshops conducted with subject matter experts and potential users to evaluate the support functions, highlight the function that appeared most promising and provide a first draft of an interface design. By emphasizing the need for user-centered solutions to complement technical advancements, we hope to contribute to safer and more effective use of LLMs.