A growing trend in assisted living involves the use of machine learning techniques to recognize hand gesture patterns tailored for people with disabilities. However, in recognition systems, the limited availability of sign language data gives rise to both data scarcity and privacy concerns. In this paper, a federated deep learning architecture is proposed for Arabic sign language recognition, aimed at addressing the challenge of deciphering the meanings conveyed by image-based hand gestures. The distributed client-server paradigm of federated learning is based on a distributed stochastic gradient descent (SGD) optimizer with a federated averaging mechanism. In addition, an interactive user interface is designed to manage distributed learning on smartphones through a client-server model, where several edge nodes collaborate to jointly learn the discriminating features of confidential data without breaching its privacy. This interactive interface improves accessibility for people with deafness or impairment using image gestures and navigation panels. Several deep learning backbones were employed in the procedure of transfer learning to fit the optimum fine-tuning setup while maximizing recognition accuracy. The experimental results show the effectiveness of the proposed FL model, achieving an accuracy of 98.80% by the FL-VGG19 configuration. There is also a significant reduction in the number of training epochs and rounds, resulting in a decrease in the computational complexity for convergence. This demonstrates the model’s capabilities in recognizing Arabic sign language and improving the communication experience for people with disabilities.