Hindi MWE Detection by Learning Phraseology from Corpora

Multiword expressions (MWEs) are linguistic structures found in many languages that span traditional word boundaries. Handling MWE presents many challenges in NLP applications. This article discusses the statistical approaches used to detect Hindi MWE in the investigation. T-scores, Pearson's-square test, log-likelihood ratios, and pointwise mutual information are employed. These statistical techniques play a crucial role in analyzing the linguistic data and determining the presence of MWEs in the context of the Hindi language. The study aims to evaluate and compare the effectiveness of these statistical measures in identifying MWEs, providing a comprehensive understanding of their applicability and performance in the specific linguistic context of Hindi. The research proposed extraction strategies for MWEs that combine syntactical idiosyncrasy (i.e., adhering to the structure of specific language patterns) and statistical idiosyncrasy (i.e., the association between constituent words of MWEs is different from regular expressions). The proposed technique is evaluated using the HDTB Treebank dataset. The experiment results show that the proposed methodology is valuable and viable, and it helps to create a blueprint that demonstrates how well it can work with the current procedures. The results indicate that all association measures can automatically extract Hindi N–N and N–V collocations as MWEs. In our experiments, the Pointwise Mutual Information (PMI) has proven more dependable than the others. The Chi-Square test also supports the findings of PMI.