End to End Unsupervised Learning-Based Endoscopic View Expansion

Endoscopic view limitation is a common issue in clinical surgery. This study proposes an end-to-end unsupervised deep learning network for endoscopic view expansion to address this issue. The network includes two sub-networks: an alignment sub-network that calculates the local correlation between image regions based on feature maps, regresses the relative offset between image vertices based on correlation, and solves homography transformation that can align endoscopic images; and a fusion sub-network that reconstructs panoramic images with multi-scale features while preserving image structural information and eliminating artifacts. The experiment shows that the proposed network can form high-quality endoscopic panoramic images under low texture, viewpoint and depth of field changes, and tissue deformation. By expanding the field of vision, the area of vision expansion near the main anatomical structures can reach more than 150%. With this technology, it is not only convenient for real-time monitoring and guidance during surgery but also for medical image processing and analysis, which can better help doctors diagnose and treat diseases.