In this paper, we introduce CrimeKGQA, an investigative system that integrates Large Language Models (LLMs) with an improved Crime Knowledge Graph (KG) to enhance the accuracy and speed of crime investigation. Traditional LLMs are often prone to producing inaccurate responses when challenged with specialized domain knowledge (e.g., sensitive crime data), as such data typically belongs to specialized and privacy-sensitive domains beyond what is included in their pre-training datasets. This can have dire consequences in some critical fields, such as crime investigation. To address this issue, we create the CrimeKGQA system, which combines the Retrieval Augmented Generation (RAG) framework with the Neo4j graph database. With this, CrimeKGQA generates precise Cypher queries on the crime investigation knowledge graph stored in Neo4j as well as dynamically retrieves and incorporates the corresponding information to deliver high-quality and contextually grounded answers. To the best of our knowledge, this is the first work that aims to exploit the potential of large language models within a crime knowledge graph to build efficient and accurate functionality for crime investigation. The experimental results show that the CrimeKGQA system can assist in answering questions and aid the investigative process, validating the efficacy of our proposed method.