Human Activity Recognition (HAR) has been a very dominant domain in pervasive computing and was used to support healthcare monitoring, fitness tracking, and smart environments. Nevertheless, models trained on specific people often malfunction when used on new individuals because actions are not precisely performed differently. To tackle this problem, we present a transfer learning framework geared towards improving the generalization ability of the HAR models for a more far-reaching base of users. To address this problem, we use a publicly available multi-user HAR dataset and design the deep learning architecture of convolutive neural networks (CNNs), and long short-term memory (LSTM) networks to learn spatial temporal patterns from sensor data. Firstly, the model is trained in a source group of users, then it is fine-tuned on some subset of target users with limited (or not for some) labeled data. Experimental results show that our method yields a substantial performance gain when deployed on new users, to up to 12% better than baseline models. The results presented herein illustrate the feasibility of transfer learning in constructing scalable and user independent HAR systems enabling their deployment in the real world.