<p>Word embeddings, while essential in natural language processing (NLP), lack a clear theoretical statistical foundation. It is challenging to interpret the specific quantity being optimized by the various training methods, or to understand the rationale behind their effectiveness in generating high-quality word representations. In this study, we aim to bridge the gap between word embeddings and statistical methodology by demonstrating that popular NLP training methods, such as Word2Vec and fastText, can be seen as statistical estimates of the pointwise mutual information (PMI) matrix; a more interpretable and consistent text vectorization method. To support this interpretation, we examined and compared the performance of PMI matrix representation methods on text semantic classification task with Word2Vec methods. We also included a proposed Variational Bayesian Inference approach to enhance the low-rank estimation of the sparse PMI matrix in word embedding tasks, and conducted a comparison of the proposed approach against the classic Shifted Positive PMI with Singular Value Decomposition (SPPMI-SVD) method. Our results demonstrate the effectiveness of PMI representation of word embedding models in real-world information retrieval scenarios, and improvement of the proposed empirical positive PMI matrix compared to the performance of the classic SPPMI-SVD method.</p>