Large Language Models (LLMs) demonstrate impressive capabilities in Question Answering tasks, yet previous research suggests current approaches fail to accurately reflect their real SPARQL query generation capabilities due to memorization effects caused by integrating benchmark datasets into the training data. These effects artificially inflate perceived performance, i.e., the quality of the showing by LLMs cannot actually be achieved on previously unseen queries. This paper presents DynBench – a novel approach to creating high-quality benchmarking datasets that address this challenge in the field of Knowledge Graph Question Answering (KGQA). We develop new datasets based on two datasets – QALD-9-Plus and LC-QuAD – by systematically replacing entities in SPARQL queries with alternatives retrieved from Wikidata and within the corresponding natural-language questions. Our findings confirm that the proposed approach successfully creates new benchmark datasets that can be used for evaluating KGQA systems. Therefore, our approach drastically reduces the risk of memorization effects, thereby increasing the trustworthiness of benchmark results for LLM-based KGQA approaches.