<p>Nanopore direct RNA sequencing offers a versatile approach for detecting multiple types of RNA modifications at a single-base resolution. In this study, we systematically evaluate 86 computational tools for detecting six RNA modifications (m⁶A, Ψ, m⁵C, A-to-I editing, m⁷G and m¹A) using direct RNA sequencing data from both RNA002 and RNA004 chemistries. We demonstrate that retraining tools with a combination of in vitro transcription and real biological samples notably enhances both accuracy and generalizability over their original implementations, especially for Ψ, m⁵C and A-to-I. Evaluations on real biological samples reveal that while m⁶A detection tools generally achieve high accuracy, non-m⁶A tools struggle with precision–recall balance, quantification accuracy and biological validity. Our findings highlight the importance of incorporating diverse training data and stress the need for tools capable of reliably distinguishing between modification types at single-base resolution. These insights provide a foundation for advancing RNA modification detection.</p>