Unstructured Text Classification Using NLP and LSTM Algorithms

In the last two decades, cancer has found as the most severe disease that causes a large number of deaths worldwide. However, it seems that the medical data are in the unstructured form, which makes it challenging for the pathologist to classify the disease in the beginning stages. This article provides a novel technique using natural language processing (NLP) for the classification of three common cancers thyroid, lung, and colon from the unstructured medical data, removing at-stop words, and articles such as ‘a,’ ‘an,’ and ‘the’ to form tokens. Tokenization is performed on larger text to divide it into smaller parts with the help of the TensorFlow technique. Besides that, long short-term memory networks (LSTM) model has been proposed to handle long sentences, especially in sequence prediction problems. This article also discusses the basic workflow of model design and prediction of three cancers. Finally, a confusion_matrix has been drawn to evaluate our model performance and found 99.41% at classifying three individual cancers.