Objectives <p>To evaluate the effects of a novel helical ultrasonic scaler tip (HUST) on dentin surface roughness and compare its performance with a non-helical ultrasonic scaler tip (NHUST) and a conventional hand scaler (Gracey curette, GC). The HUST was designed to improve calculus removal efficiency while minimizing dentin surface damage.</p> Materials and methods <p>This in vitro experimental study used standardized dentin blocks prepared from extracted human teeth to evaluate the effects of three instruments (HUST, NHUST, and GC). Surface roughness was quantified by measuring the total height of the profile (Pt) using a non-contact surface profilometer based on coherence correlation interferometry. Tip wear was assessed qualitatively using scanning electron microscopy. The sample size was determined based on preliminary experimental observations. Statistical analyses included the Shapiro–Wilk test for normality, followed by the Kruskal–Wallis test and Steel–Dwass post hoc comparisons.</p> Results <p>No visible wear was observed on either ultrasonic tip, even after 400 applications. Pt was significantly higher in the NHUST group (18.80 ± 6.21&#xa0;μm) than in the HUST (10.26 ± 4.31&#xa0;μm) and GC (6.05 ± 3.78&#xa0;μm) groups (<i>p</i> &lt; 0.01). No significant difference was observed between the HUST and GC groups. All experimental groups showed greater roughness than the polished control (3.29 ± 1.14&#xa0;μm), but the HUST group exhibited significantly lower surface alteration than the NHUST group.</p> Conclusions <p>The HUST resulted in lower dentin surface roughness than the NHUST.</p> Clinical relevance <p>The HUST may contribute to safer ultrasonic instrumentation with reduced dentin surface alterations.</p> Graphical abstract <p></p>