Multi-view breast ultrasound imaging has been routinely performed in clinical settings to ensure comprehensive disease evaluation. Recently, artificial intelligence (AI) has been developed to interpret medical images; however, most of the current AI models are restricted to single-view images, resulting in weak representation of breast 3D tissues. Here, we develop an Image Representation and Unified learning Method (IRUM) on a dataset comprising 3800 ultrasound images from 1900 patients with an accuracy of 86.8%. Owing to the design of four distinct learning modules, the proposed IRUM is not only able to predict breast cancer risk using multi-view inputs, but also compatible with single-view input (a commonly encountered situation in clinical practice). We demonstrate that the IRUM achieves superior performance to conventional single-view and multi-view approaches to a certain degree.