We present a time of flight (TOF) mixed reality (MR) digital twin mobile system supporting three-dimensional (3D) tracking and defect detection using recursively fused multimodal segmentation paradigm. Simplified machine learning can be used for clustering multimodal 3D semantic label distribution (output of generic data trained segmentation deep learning model) and to reduce the need to obtain high cost and extremely scarce non-generic training data to flexibly customize segmentation for non-generic enterprise defect inspection applications. The fused model first segments with 3D physics properties (reflection, curvature, materials etc.) obtained from TOF and tracks objects with defect from a 3D scene and then further segments recursively on different level of details to detect defects with quantification analysis based on segmentation distribution statistic distance. This method also removes the need to do compute intensive non-real-time algorithms (3D mesh generation, SLAM bundle adjustment and cross source 3D alignment) needed for 3D defect detection. User can do portable free hand acquisition to track and quantify the severity of 3D anomaly defects and categories of 3D configuration without the need to follow strict data capture guidance and 3D point cloud alignment registration as required by other state of the art enterprise MR systems.