Knowledge base question answering (KBQA) is an open and challenging task, which aims to provide knowledge base (KB) answers according to natural language questions. Most existing supervised KBQA methods require substantial labeled data along with a customized training framework to achieve satisfactory performance, due to the complexity of KBs and diversity of questions. Additionally, supervised KBQA methods are usually KB-specific, thus poor at generalization. Owing to the impressive in-context learning capability of large language models (LLMs), as well as their transformation ability from natural language to structured code, this paper adopts LLMs to achieve few-shot logical expression (LE) based KBQA. Considering the complexity of directly-generating LEs, we propose to decompose the direct LE-generation by LLMs into logical skeleton generation and expression component discrimination. Furthermore, we transform the generative task of LLMs into the candidate selection of skeleton and component by leveraging LLMs’ discriminative instead of generative ability. To alleviate LLMs’ hallucination, we further filter irrelevant candidates to the given question to enhance the qualities of in-context learning samples. We conduct extensive experiments on popular KBQA datasets and achieve substantial few-shot performance improvements over the state-of-the-art LLMs-grounded method.