<p>High-frequency mechanical oscillators with long coherence times are essential for realizing a variety of high-fidelity quantum sensors, transducers and memories. However, the coherence times needed for quantum applications require advances in probing and mitigating the origins of phonon decoherence in the materials used for mechanical oscillators. Here we identify key sources of phonon decoherence in crystalline media by combining non-invasive laser spectroscopy with materials analysis. Using micro-fabricated high-overtone bulk acoustic-wave resonators as an experimental test bed, we find that phonon–surface interactions are the dominant source of phonon decoherence in crystalline quartz. The probable causes are lattice distortion, subsurface damage and a high concentration of elemental impurities. We use an optimized polishing process to remove the compromised surface layer and produce resonators with quality factors exceeding 240 million at 12 GHz, corresponding to phonon coherence times of over 6 ms. We verify that these mechanical oscillators have negligible dephasing. Building on these results, we propose a path that could reach coherence times beyond 100 ms as the basis for high-frequency quantum memories. These findings demonstrate that enhanced control over surfaces enables a substantial reduction in dissipation and noise.</p>