The COVID-19 pandemic has highlighted the critical need for efficient and objective severity assessment methods. Current approaches relying on manual interpretation of lung ultrasound (LUS) images are limited by subjectivity and inefficiency, while existing computer-aided techniques either depend on segmentation annotations or perform only basic image-level analysis. To address these limitations, we developed a novel framework that systematically combines LUS data with clinical information through a two-stage approach. The first stage employs a soft ordinal regression distance (SORD) approach to model the progressive relationship between image features and pathological severity at the LUS image level. The second stage introduces a dynamic gated multi-instance learning (DG-MIL) module that comprehensively integrates features across all lung zones while simultaneously encoding clinical data for unified patient-level assessment. Validated on a dataset of 238 training cases and 80 test cases, our method achieves 72.25% accuracy for four-way classification and 89.13% accuracy for binary severe/non-severe assessment. The segmentation-free design enhances practical applicability and facilitates large-scale deployment, while the radiation-free nature of ultrasound imaging makes this approach particularly valuable for vulnerable populations. These results demonstrate the framework's potential to improve COVID-19 management workflows through more efficient severity assessment.