In this study, an end-to-end deep learning method is proposed to improve the accuracy of continuum estimation in low-resolution gamma-ray spectra. A novel process for generating the theoretical continuum of a simulated spectrum is established, and a convolutional neural network consisting of 51 layers and more than \(10^5\) parameters is constructed to directly predict the entire continuum from the extracted global spectrum features. For testing, an in-house NaI-type whole-body counter is used, and \(10^6\) training spectrum samples (20% of which are reserved for testing) are generated using Monte Carlo simulations. In addition, the existing fitting, step-type, and peak erosion methods are selected for comparison. The proposed method exhibits excellent performance, as evidenced by its activity error distribution and the smallest mean activity error of 1.5% among the evaluated methods. Additionally, a validation experiment is performed using a whole-body counter to analyze a human physical phantom containing four radionuclides. The largest activity error of the proposed method is \(-5.1\%\) , which is considerably smaller than those of the comparative methods, confirming the test results. The multiscale feature extraction and nonlinear relation modeling in the proposed method establish a novel approach for accurate and convenient continuum estimation in a low-resolution gamma-ray spectrum. Thus, the proposed method is promising for accurate quantitative radioactivity analysis in practical applications.