Building upon the Semantic and Spatial Matching for Visual Place Recognition (SSM-VPR) model for Waterborne Imagery, we develop a novel pipeline that leverages segmentation at multiple stages in order to enhance performance, filter out non-discriminative features, and to limit spatial matching to relevant class based edge lines. Our approach is motivated by the unique nature of waterborne imagery, where salient land features often make up a minority of the overall image, with the rest being non-discriminative sea and sky. In order to measure improvements in performance, speed, and storage space We apply each individual segmentation-aware method to the pipeline in, before fusing these methods into a single highly robust and efficient pipeline that balances the benefits of each. We evaluate this approach on a waterborne image dataset and compare our novel approach to the original SSM-VPR model, showing improvements in precision versus recall, total recall, and providing efficient inference time for real-world application.