A Sentiment Analysis of Tweets by Using TF-IDF Vectorizer and Lemmatization with POS Tagging

The rapid development of the Internet, social media, and data preprocessing approaches has led to major demands for new technologies, including data analysis. The primary goal of sentiment analysis is to find the sentiment behind the user reviews being given across various platforms. In this regard, we start the discussion on sentiment analysis research and then describe some of the recent works within this domain. This chapter is presented in two stages. For the first stage, we have considered two datasets, consisting of 16 lakh tweets and nearly 6 lakh Amazon reviews (data entities), and they contain six columns, each of which describes the features of our data entities. Because the data were originally unstructured, we subject them to data processing and applied classification models. The second stage of our project includes arrangement, development, and analysis. To normalize the data by subjecting them to parts of speech (POS) tagging and lemmatization; learning algorithms, like linear support vector machine (SVM) classification, Bernoulli Naïve Bayes, and the logistic regression model, are used. Accuracy is measured to assess the performance of the system, via POS tagging+ lemmatization + TF-IDF vectorizer. Logistic regression had the highest accuracy, at 81%, for Twitter, and linear SVM had an accuracy value of 91%, according to Amazon’s data.