Falls are often attributed to poor muscle function, with weak hand grip strength clinically recognized as a major risk factor. However, grip strength is rarely assessed clinically. Low radiation dual-energy X-ray absorptiometry (DXA) whole-body scans, that can be obtained during routine osteoporosis screening, offer a comprehensive overview of body composition, thereby providing valuable information for musculoskeletal health. Here, we propose a machine learning technique, exploiting image and clinical data to classify weak grip strength (<22 kg), thereby enhancing fall prediction capabilities. To effectively utilize both discrete and continuous grip strength information, we introduce a novel Supervised Contrastive learning (SupCon) loss strategy, supplemented by regression loss guidance. Additionally, we present a pipeline featuring a unique Region of Interest (RoI) extraction strategy in the data preprocessing procedure, which is designed to focus on areas of genuine interest. Our proposed multi-modal contrastive learning (MMCL) framework enhances feature separability, and class diversity in the latent space, by leveraging different types of information. We evaluate the performance of our framework using a dataset of older women (2144 images); and employ survival analysis for evaluating future fall-related hospitalization risk over 5 years. Our results demonstrate that weak grip strength classified by the proposed approach achieves high sensitivity and accuracy and predicts risk of injurious falls in older women.