Quantifying Similarity: Text-Mining Approaches to Evaluate ChatGPT and Google Bard Content in Relation to BioMedical Literature

The emergence of generative AI tools, empowered by Large Language Models (LLMs), has shown power in generating content. The assessment of the usefulness of such content has become an interesting research question. Using prompt engineering, we assess the similarity of such contents to real literature produced by scientists. In this exploratory analysis, we prompt-engineer ChatGPT and Google Bard to generate clinical content to be compared with medical literature, and we assess the similarities of the generated contents by comparing them with biomedical literature. Our approach is to use text-mining methods to compare documents and bigrams and to use network analysis to check the centrality. The experiments demonstrated that ChatGPT outperformed Google Bard in different similarity and term network centrality methods, but both tools achieved good results compared to the baseline.