<p>We live in the era of big data, where many applications involve massive data that are often too large for storage, pre-processing, and analysis on a single computer. To address these challenges, data is often distributed across multiple computers, and distributed learning, inspired by the divide-and-conquer principle, has emerged as a popular approach for managing such enormous data. This paper focuses on fitting sparse linear regression models within a distributed statistical learning framework. We investigate the performance of a class of shrinkage estimators, commonly referred to as pretest and Stein-type estimators, in a distributed computing environment. These estimators are particularly useful for sparse estimation in the presence of multicollinearity when the data is divided into smaller subsets (samples) across multiple machines. The estimators are linear combinations of the sub-model and full-model estimators, designed to induce sparsity while effectively managing the bias-variance trade-off on each local machine. These advantages are inherited by our aggregated estimator, called one-shot distributed estimator, which combines local estimators calculated on individual machines. We establish consistency and asymptotic normality of the proposed one-shot estimators, supporting our findings with simulations and analysis of the Million Song Year Prediction Dataset.</p>