In 2017, the National Assessment of Educational Progress (NAEP) transitioned the reading and mathematics assessments at grades 4 and 8 to a digitally-based platform. As part of the transition, a comprehensive mode evaluation study was conducted to provide evidence on whether and how to maintain trend reporting across the paper-based and digitally based administrations. The study involved administering the NAEP instruments in paper and digital formats to randomly equivalent groups. Although items retained many of their unique properties, such as approximate rank-order difficulty, items were generally more difficult when administered in digital format. Apart from constructed response items in grade 4 showing a larger increase in difficulty when adapted to digital format, the increased difficulty was fairly uniform across all items within a subject and grade, and almost all item mean scores were significantly different between modes. To account for differences in the measurement instruments, common population linking was applied. After linking, few significant differences in mean estimates for state/jurisdictions, districts, and subpopulations were found between the digital and paper results. Observed significant differences were associated with small effect sizes, and the effect sizes were inconsistent across grades and across subjects.