Two-stage retrieval systems, consisting of a fast first-stage retriever and a powerful second-stage re-ranker, are a standard architecture for high-performance information retrieval. The recent advent of Large Language Models (LLMs) has created an opportunity to leverage their advanced reasoning capabilities as powerful zero-shot re-rankers, yet the principles governing their effectiveness and scalability remain underexplored. In this paper, we introduce FreeRanker, a novel zero-shot re-ranking framework inspired by the philosophy of latent planning. FreeRanker employs an LLM as a “planner” to perform explicit reasoning over a set of candidate documents, culminating in the selection of a single, most relevant document. We conducted a comprehensive empirical study on the SQuAD dataset to evaluate the performance of FreeRanker, systematically examining how its effectiveness scales with the size and architecture of various open-source LLMs. This work provides insights into the behavior of LLMs as zero-shot re-rankers and analyzes the factors that contribute to their performance in this critical information retrieval task.