<p>Although the ability to comprehend what one is reading is one of the most fundamental necessities to function within society, the reading comprehension skills of students have recently been on the decline in many countries. An essential prerequisite to reading comprehension is the ability to read fluently, which is defined as the ability to read (aloud) with accuracy, speed, automaticity and prosody. Current oral reading fluency assessment instruments seldom provide detailed diagnostics however, and bestow a heavy testing burden on practitioners. Recent developments in Artificial Intelligence-based assessment methodology might provide a solution to current assessment issues, but thorough validations of such procedures have proven scarce. This study evaluates whether valid word decoding and passage reading measures (accuracy, speed and automaticity) can be generated for a semi-transparent language, using an automatic speech recognition (ASR) based oral reading fluency assessment instrument. A validation study was conducted, using the Argument-Based Approach to Validation. Data concerned 176 h of speech data, and the results of 569 and 622 oral word- and passage reading tests that are currently administered in primary schools, from 653 children attending the second- or third grade of Dutch primary education. The results of the validation indicate that it is possible to generate fluency metrics for a semi-transparent language, using an ASR-based oral reading fluency assessment instrument. Future researchers are advised to further optimize the ASR, evaluate its errors, and realize a prosody component, completing the envisioned reading fluency assessment instrument, thereby improving reading fluency assessment throughout primary education.</p>