It is significantly challenging to recognize daily human actions in domestic settings due to the diversity and dynamic changes in unconstrained home environments. It spurs the need to continually adapt to various users and scenes. Fine-tuning current video understanding models on newly encountered domains often leads to catastrophic forgetting, where the models lose their ability to perform well on previously learned scenarios. To address this issue, we formalize the task of Video Domain Incremental Learning (VDIL), which enables continual learning across domain shifts while maintaining a fixed set of action classes. While most continual learning research focuses on class-incremental settings, domain-incremental learning remains underexplored in video understanding. In this work, we introduce a benchmark for domain-incremental human action recognition in dynamic home settings, comprising three domain splits: user-based, scene-based, and hybrid. We also propose a simple yet effective baseline that combines replay and reservoir sampling without access to domain labels, designed for memory-constrained, task-agnostic scenarios. Extensive experiments show that our approach consistently outperforms existing continual learning methods across all benchmark.