Text Standardization in Public Salary Disclosure Data
摘要
This paper presents an automated methodology for clustering text entities within salary datasets, in particular employer names. This is a crucial step for data standardization and subsequent robust regression analysis, particularly when investigating employer- and job-specific effects. The approach leverages the power of transformer models (MPNer-base, MiniLM-L6 and RoBERTa-large) to generate semantically rich vectorial representations (embeddings) from diverse text descriptions of employer names in the data. These high-dimensional embeddings then serve as input for subsequent clustering algorithms, which group together similar descriptions (e.g., “University of Guelph” and “Guelph University” for employer). The identified clusters enable the standardization and cleaning of employer. We illustrate the effect of clustering employer data in a regression model framework applied to a public salary disclosure data for Ontario, Canada.