We present an annotation-free, multimodal approach to predict risk in the setting of osteoporosis-induced fractures, and to attribute this risk to specific vertebrae. Moreover, we demonstrate that using low-dose spine X-rays is sufficient to predict risk, but that predictions are drastically improved by including only image patches of vertebral bodies. Using visual explainability methods, risk can be attributed to individual vertebrae to increase interpretability of model decision making. We validate the results across multiple types fracture events using common evaluation metrics from survival analysis, such as the C-index and Brier score. Our approach shows significant improvements over the clinical baseline, and we demonstrate that the model may be used to identify high-risk patients, reinforcing potential clinical utility.