Advancements in Generative text models, specifically Large Language Models (LLMs), raise concerns about the extent to which such models perpetuate biases and prejudices through their responses. However, in spite of this concern, research in bias and fairness for generative text models remains sparse and focused around notions of group fairness. In this work, we aim at formalising a notion of fairness for LLM inspired by individual fairness – i.e., that similar sentences are treated similarly by the model. A fairness criterion is then defined which encodes the notion of individual fairness by specifying that the distance between the responses for a particular generative text model given some input prompts should be no greater than the distance between the respective input prompts, where the distance is quantified using an LLM-tailored similarity metric. This fairness criterion is therefore incorporated into existing dataset-based methods for identifying biases in NLP models using exhaustive testing. Evaluating our methodology against two state-of-the-art generative text models (GPT-2 and BlenderBot) with known biases using two different similarity metrics, the results offer positive evidence for incorporating additional context, through a similarity metric, into methods for evaluating fairness in generative text models.