Hepatocellular carcinoma (HCC) is a highly lethal cancer in which transarterial chemoembolization (TACE) is widely applied in patients with HCC, regardless of whether they are in the early or late stage. However, a notable portion of patients do not benefit from TACE, which is often accompanied by adverse side effects. Therefore, selecting the right patients for TACE before surgery is essential, and having an accurate model to predict the response to TACE is highly valuable. This paper proposes Attention-based Fully Cross-Scale Convolutional Networks (ACSCN), which are primarily composed of Cross-Scale Blocks (CB), Channel Attention Mechanism (CAM) modules, and a Phase Attention Mechanism (PAM) module. CB obtains multi-scale information on HCC tumors. The CAM module assigns weights to feature information across channels at various scales to enhance the recognition of critical channel features. Additionally, the PAM module was designed to enhance the capture of tumor-specific texture characteristics in multi-phase contrast-enhanced MRI by integrating features from different MRI phases. The efficacy of ACSCN is evaluated by comparing it with existing classic networks using a liver cancer dataset (154 cases). The effectiveness of the CAM and PAM designs is validated through ablation experiments. ACSCN’s strong performance indicates its potential as an effective aid for accurate clinical management and informed decision-making.