Intelligent Mobile Product Recognition for Augmented Reality in Smart Shopping

Many shop owners struggle to digitalize their physical offerings to achieve business continuity in times of shop closure, and to achieve higher sales with shoppers around. Due to business dynamics, they can also often not rely on an Enterprise Resource Planning (ERP) system to retrieve digitalized product information, and thus need to start the digitalization from scratch. A purely optical digitalization is promising due to being cost-efficient in its implementation, and enabling further functionality such as Augmented Reality (AR) for increased customer engagement. This paper discussed three optical approaches to help shop owners digitalize their products based on images and videos and develop digital channels such as e-commerce websites and AR application on that basis. The first approach uses Common Objects in Context (COCO) Single Shot Detector (SSD) Transfer Learning to identify the products from the shelf via real-time video stream and a user-friendly website. The second approach offers a Mask R-CNN (Region-Based Convolutional Neural Network) model to identify the objects and compare them in the market database using the Structural Similarity Index Measure (SSIM). The last approach implements an Optical Character Recognition (OCR) model to detect and recognize the label information from the taken image of the product shelf.