Decoding Named Entities: Analysing Hindi–English Code-Mixed Social Media Text

Named Entity Recognition (NER) holds significant importance within Natural Language Processing, especially for the extraction of valuable insights from text by detecting and categorising entities, such as individuals’ names, organisations, and geographical locations. While NER has made considerable progress in dealing with conventional text, it faces distinct challenges in code-mixed content, which is prevalent in multilingual societies and social media platforms. For example, a sentence might seamlessly blend words in English and Hindi, creating entities that require cross-lingual recognition. Researchers are continuously adapting and innovating deep learning techniques, creating specialised datasets, and exploring various methods to effectively identify and classify named entities in informal, multilingual, and code-mixed text. In this study, a comprehensive assessment of Decision Tree, Naive Bayes, Random Forest and LSTM models, was carried out over a well curated dataset tailored for the identification of Named Entities within Code-Mixed Hindi–English social media text, with a primary emphasis on tweets. Each model showcased notable performance on the provided dataset. The Decision Tree model distinguished itself with an impressive f1-score of 0.94, and Naive Bayes achieved a notable f1-score of 0.67. Both the Random Forest and LSTM models demonstrated remarkable capabilities, attaining an impressive f1-score of 0.96 and standing out as the best performers.