Background <p>The prognosis and treatment outcomes for patients with rectal cancer are critically dependent on an accurate and comprehensive preoperative evaluation.Three-dimensional endorectal ultrasound (3D-ERUS) has demonstrated high accuracy in the T staging of rectal cancer. Thus, we aimed to develop a computer-aided diagnosis (CAD) tool using a deep learning model for the preoperative T-staging of rectal cancer with 3D-ERUS.</p> Methods <p>We retrospectively analyzed the data of 216 rectal cancer patients who underwent 3D-ERUS. The patients were randomly assigned to a training cohort (<i>n</i> = 156) or a testing cohort (<i>n</i> = 60). Radiologists interpreted the 3D-ERUS images of the testing cohort with and without the CAD tool. The diagnostic performance of the CAD tool and its impact on the radiologists’ interpretations were evaluated.</p> Results <p>The CAD tool demonstrated high diagnostic efficacy for rectal cancer tumors of all T stages, with the best diagnostic performance achieved for T1-stage tumors (AUC, 0.85; 95% CI, 0.73–0.93). With assistance from the CAD tool, the AUC for T1 tumors improved from 0.76 (95% CI, 0.63–0.86) to 0.80 (95% CI, 0.68–0.94) (<i>P</i> = 0.020) for junior radiologist 2. For junior radiologist 1, the AUC improved from 0.61 (95% CI, 0.48–0.73) to 0.79 (95% CI, 0.66–0.88) <i>(P</i> = 0.013) for T2 tumors and from 0.73 (95% CI, 0.60–0.84) to 0.84 (95% CI, 0.72–0.92) (<i>P</i> = 0.038) for T3 tumors. The diagnostic consistency (κ value) also improved from 0.31 to 0.64 (<i>P</i> = 0.005) for the junior radiologists and from 0.52 to 0.66 (<i>P</i> = 0.005) for the senior radiologists.</p> Conclusion <p>A CAD tool utilizing a deep learning model based on 3D-ERUS images showed strong performance in T staging rectal cancer. This tool could improve the performance of and consistency between radiologists in preoperatively assessing rectal cancer patients.</p>