The rapid advancements in transformer-based language models have opened new possibilities for applications such as question answering and synthetic dataset generation. However, generating high-quality synthetic datasets can be computationally expensive and require significant resources. Fine-tuning a language model on a synthetic dataset can lead to improved performance on context-dependent questions, but it is crucial to evaluate the quality of these datasets beforehand. This study explores the benchmarking of synthetic natural language datasets generated with open-source language models for question answering applications. We fine-tune the same base model on each dataset independently and assess its performance on common benchmarks for question answering with context. Additionally, we examine metrics of the synthetic datasets themselves to identify potential indicators of downstream benchmark performance. Our results show that measuring the semantic similarity between questions is an indicator of domain diversity in a synthetic question-and-answer dataset. We also demonstrate that short answer lengths may indicate low quality chain-of-thought answers. Overall, Llama 3 8B Instruct performs most consistently well, of the models considered, in synthetic dataset generation. Our findings provide valuable insights for practitioners seeking to develop effective synthetic datasets for language-based applications.