With rich text descriptions and structural information, text-attributed graphs (TAGs) are widely utilized across various applications such as academic networks, social networks, recommendation systems and knowledge engines. In these applications, the task of information retrieval is pivotal for interacting with TAGs. Considering the remarkable and versatile capabilities of large language models (LLMs), we investigate into unleashing LLMs for retrieval on TAGs. However, it is still an open research question in understanding and leveraging structural patterns with LLMs for effective search on graph nodes and structures. In this paper, we propose an LLMs-empowered architecture for TAG retrieval named Query Subgraph Schema Guided Retrieval (QSS-Ret). Our method begins with the generation of a query subgraph schema by LLMs, serving to distill and identify key information from the retrieval query in a graphical form. Based on this schema, we retrieve query-relevant subgraphs by locating candidate nodes and adaptable structures under a Depth First Strategy (DFS)-based strategy. To attend to the fine-grained textual information associated with graph nodes, the extracted subgraphs are summarized and then consumed by the LLM for the re-ranking of candidate graph nodes. Empirical evaluations demonstrate that our method effectively enhances structural understanding capabilities across TAG retrieval benchmarks from different domains. Our code can be found at https://github.com/yifeiHuang623/QSS-Ret/.