Prompt engineering and chain of thought prompting have resulted in improvements in performance on large language model benchmark tests, but the metrics from benchmarking tests are limited, missing qualitative and complex errors that tend to occur in LLM generated explanations of reasoning. Automated methods remain unable to perform detailed output analysis. A framework for detailed output analysis, based on Toulmin’s model of argumentation, is proposed in this work. This framework is suitable for manual evaluation of LLM outputs and prompting methodologies, and provides insights into the qualities of the explanations generated. This approach is demonstrated using the Mistral 7B Instruct model along with zero shot, few shot, zero shot chain of thought, chain of thought, and self-consistency chain of thought prompting methodologies. Case studies are presented showing the types of insights possible, including the ability to identify hallucinations and logical inconsistencies overlooked by automated evaluations. Observations are made on the relationship between the quality of LLM generated explanations of reasoning and the correctness of that reasoning. These observations show how manual evaluation of LLM outputs using the framework of Toulmin’s model results in the identification of failures and inconsistencies that are not able to be identified with automated evaluation approaches. This Toulmin model based approach provides a new evaluation methodology that delivers fine grained insights suitable for guiding future model development, prompt design, and evaluation of the application specific suitability of prompting methodologies.