Structure from Motion (SfM) has been widely studied in many fields, such as computer vision, photogrammetry, robotics, etc. Recent advancements focus on improving the real-time performance of SfM, which is crucial for applications in augmented reality, mixed reality, robotics, etc. However, the robustness of real-time processing is still limited by outliers in the feature extraction and matching process, stemming from challenging scenes depicting objects with poor texture, repetitive structures, and symmetric objects, which can cause blunders in the view-graph. Focusing on these scenes, a Learning-based View-Graph generation method (LVG-SfM) is investigated and integrated into the on-the-fly SfM pipeline [43]. First, to provide a higher number of reliable matches and generate a more robust view-graph, a set of SoTA learning-based feature extraction and matching methods [19] are tested. Then, the spuriously incorrect two-view geometries generated from repetitive structures are removed from the view-graph with the help of SoTA learning-based disambiguation network - Doppelgangers [3]. Experimental results demonstrate that our LVG-SfM can successfully work on-the-fly on challenging ambiguous scenes with poor textures and repetitive structures, achieving correct scene reconstructions and robustifying SfM. Project website at: https://sygant.github.io/lvgsfm .