Sign languages are part of the natural way of communication of the hearing-impaired community, allowing for efficient expression of thoughts and feelings. Tamil Sign Language (TSL) used by the deaf Tamil-speaking community lacks technological support and hence a huge communication gap looms large in day-to-day interactions, in the classroom and at work. Traditional methods of interpretation are largely based on human interpreters whose presence is not always guaranteed. Despite the progress made in sign language recognition, a number of challenges still remain, especially due to the unavailability of public datasets for TSL and the insufficiency of adequate research on bidirectional translation compared to the common studies on established sign languages like ASL and BSL. Current approaches tend to address sign-to-text translation alone or fail to identify intricate hand movements accurately and differentiate between gestures with slight differences. The solutions to these challenges demand an innovative strategy that incorporates deep learning, natural language processing and 3D animation to facilitate smooth communication between TSL users and non-sign language users. To address these challenges, we propose a Bidirectional Translation System for TSL Recognition that can perform Sign-to-Speech and Speech-to-Sign translation. For Sign-to-Speech, we benchmarked different deep learning models to identify the best-performing model for gesture recognition. Identified gestures are then analyzed by natural language processing for meaningful sentence construction and converted into natural speech. For Speech-to-Sign, spoken Tamil is processed to identify key information, which is translated to TSL using a 3D animated avatar. Initial findings indicate improved recognition accuracy, speed of translation, and ease of use compared to existing approaches. This research adds to UN Sustainable Development Goal 10 – Reduced Inequalities, with a promising step towards inclusive communication for the hearing-impaired population.