Dual-Stream Context-Aware Neural Network for Survival Prediction from Whole Slide Images

Whole slide images (WSI) encompass a wealth of information about the tumor micro-environment, which holds prognostic value for patients’ survival. While significant progress has been made in predicting patients’ survival risks from WSI, existing studies often overlook the importance of incorporating multi-resolution and multi-scale histological image features, as well as their interactions, in the prediction process. This paper introduces the dual-stream context-aware (DSCA) model, which aims to enhance survival risk prediction by leveraging multi-resolution histological images and multi-scale feature maps, along with their contextual information. The DSCA model comprises three prediction branches: two ResNet50 branches that learn features from multi-resolution images, and one feature fusion branch that aggregates multi-scale features by exploring their interactions. The feature fusion branch of the DSCA model incorporates a mixed attention module, which performs adaptive spatial fusion to enhance the multi-scale feature maps. Subsequently, the self-attention mechanism is developed to learn contextual and interactive information from the enhanced feature maps. The ordinal Cox loss is employed to optimize the model for generating patch-level predictions. Patient-level predictions are obtained by mean-pooling patch-level results. Experimental results conducted on colorectal cancer cohorts demonstrate that the proposed DSCA model achieves significant improvements over state-of-the-art methods in survival prognosis.