Activity recognition in smart homes has recently gained significant attention among the researchers in both computer vision and multimedia communities. Applications like video surveillance, and elderly care require a comprehensive understanding of different activities occurring in daily lives. However, this task remains relatively underexplored due to availability of only limited annotated data. In this paper, we introduce a model to first recognize fine-grained actions, and subsequently, identify composite activities resulting from such actions, in smart home environments. An Attention-based Shallow Residual Network (ASRNet) is applied first for feature extraction, where data scarcity is explicitly addressed by utilizing multiple datasets. We then perform fine-grained action recognition with an Aquila Hunger Games Search-optimized Bidirectional Long Short-Term Memory (AHS-BiLSTM) network. We then use the detected actions to identify composite activities. We construct an Activity Knowledge Graph (AKG), a weighted undirected graph, from the annotated data. Using a Bayesian inference on AKG, we identify composite activities from the already detected fine-grained actions. Experimental results on the “MPII Cooking 2” and “Toyota Smarthome” datasets demonstrate the effectiveness of our approach.