Differentiating malignancy from liver parenchyma in Ex-Vivo OCT images using anomaly detection

Primary liver cancer and colorectal liver metastases (CRLM) pose significant challenges, because of limited early diagnosis and the reliance on time-consuming frozen section analysis during surgery to confirm complete tumor resection (R0). This study investigates the potential of optical coherence tomography (OCT) combined with anomaly detection for differentiating hepatocellular carcinoma (HCC), intrahepatic cholangiocarcinoma (iCCA) and CRLM from normal liver parenchyma, ex-vivo. Our dataset comprises 173 OCT images sourced from 69 patients undergoing liver surgery. We leveraged pre-trained neural networks with frozen weights and statistical outlier modeling to train an anomaly detection model using only non-cancer parenchyma scans. Given the small-scale nature of the dataset and the presence of label uncertainty, a stratified cross-validation procedure was employed to robustly assess the model’s performance in accurately matching OCT scans with their corresponding histological diagnoses. This resulted in promising classification performance using a pre-trained Vision Transformer: sensitivity 80%, specificity 78%, accuracy 79%, and area under the receiving-operating-characteristic-curve (ROC-AUC) of 81%. While limited by a relatively small and noisy dataset, this study highlights the promising potential of OCT combined with anomaly detection for intraoperative liver cancer detection. This semi-supervised learning approach offers several advantages, including reduced training time and data requirements, as well as interpretable anomaly scores.