Background <p>Hepatocellular carcinoma (HCC), particularly in recurrent or treatment-refractory cases, often exhibits poor responsiveness to radiation therapy, increasing the risk of radiation-induced liver disease, necessitating innovative treatment approaches. Boric acid-mediated boron neutron capture therapy (BA-BNCT) has emerged as a promising approach for liver cancer. This study aims to improve BA-BNCT efficacy for radioresistant HCC by exploring sensitization agents, enhancing treatment while minimizing irradiation doses and side effects.</p> Methods <p>We targeted the DNA homologous recombination repair (HRR) protein RAD51. Before neutron irradiation, a RAD51 inhibitor, B02, was administered to evaluate its sensitization effect on both HepG2 and the radioresistant HepG2R cells. We examined the cell death mechanism, focusing on the expression profile of LC3B after BA-BNCT, to investigate its impact on DNA repair responses, especially on autophagy and apoptosis.</p> Results <p>We observed that inhibition of RAD51 led to increased γH2AX, the DNA double-strand break marker. Additionally, combining the RAD51 inhibitor B02 with BA-BNCT resulted in tumor cell arrest in the G₀/G₁ phase, indicating altered cell cycle regulation. In exploring cell death mechanisms, we observed increased autophagy following BNCT, potentially as a response to cellular stress induced by DNA damage in tumor cells. The combination of B02 and BA-BNCT significantly disrupted autophagic flux and promoted apoptosis in the tumor cells.</p> Conclusions <p>Combining a RAD51 inhibitor with BA-BNCT significantly enhances the anti-tumor efficacy against radioresistant HCC and parental HCC cells. This proof-of-concept study suggests that the combination treatment can achieve comparable or superior therapeutic effects using lower radiation doses, thereby reinforcing the potential of BNCT for treating recurrent HCC.</p>