Improving a Fuzzy Matching Engine

InSiteTrack requires that individual client products are matched to competitor products in order to provide pricing intelligence on their market. To increase the speed of the product matching process, string matching suggestions are put forward. The previous solution utilized the Levenshtein algorithm, but this was found to be computationally inefficient and also omitted numerous possible matches. After analysing a number of possible solutions, an algorithm that utilized Term Frequency Inverse Document Frequency (TF-IDF) with n-grams and a cosine similarity calculation seemed the most promising. The key benefits of this solution were the fact that it uses basic arithmetic when comparing two names and that it weighted the rarity of terms across a collection. A prototype system was quickly developed that was specific to a use case; with the success of the test program, a database connected version was created that has since been deployed. The resulting system is significantly faster, more accurate and more adaptable to specific datasets than the solution that it is now replacing.