Using Neural Coherence Models to Assess Discourse Coherence

Discourse coherence is an important characteristic of well-written texts and coherent speech. It is observed at several levels of discourse analysis: lexical, syntactic, semantic, and pragmatic. Recent work on discourse coherence uses deep neural network architectures to model coherence. However, most of these architectures are not linguistically explainable. In this paper, we propose a fine-tuned Large Language Model (LLM) and three interpretable approaches for modeling discourse coherence, that target different levels of discourse analysis and coherence information, capturing contextual information, semantic relatedness between adjacent sentences and paragraphs, and syntactic patterns of coherent texts. We want to determine whether these explainable approaches lead to competitive results compared to the proposed fine-tuned LLM. These architectures are evaluated on the multi-domain Grammarly Corpus of Discourse Coherence (GCDC) and compared to state-of-the-art (SOTA) and recent models. The results of our experiments show that the syntactic patterns combined with the semantic relatedness are a good indicator of the overall coherence and highlight the importance of the number of training examples for the model’s ability to use the information provided by the syntactic patterns to make accurate predictions. Furthermore, the contextual information captured by the transformer-based model achieves good results that significantly outperform all other models, showing that the use of a fine-tuned LLM is now typically the best performing approach, despite being less interpretable than other methods.