LASSO Regression and K-means Clustering: Application to Indian Labor Market
摘要
This chapter introduces fundamental machine learning (ML) concepts, including the bias-variance trade-off, regularization and cross-validation, contrasting them with traditional econometric approaches and illustrating their application in social sciences. It explores one technique from each of the supervised and unsupervised learning paradigms—LASSO regression and k-means clustering. The chapter also provides an overview of the basic principles of supervised and unsupervised learning principles while demystification commonly used terms in ML parlance such as deep learning, neural networks, random forests, recommendation systems and matrix factorization. Following the theoretical discussion, an empirical application is presented using R, based on primary data on job market outcomes and skills from 784 respondents in Bangalore, India. This application examines (i) the relationship between skill claims and job outcomes in informal work, and (ii) the clustering of skills among informal workers. The chapter concludes with a summary and outlines potential research directions for applied researchers interested in working at the intersection of ML and econometrics.