Walking direction is vital for applications such as surveillance, security, traffic safety systems and health monitoring. Silhouettes and Skeleton joint coordinates are commonly used gait representation modalities, both with their own advantages. Silhouettes are rich in representative features while skeleton coordinates are more robust to noise. In this paper, we explore the two popular modalities for walking direction estimation. We leverage temporal sequence modeling to extract gait relevant information and capture the correlation between consecutive frames and extract motion features. Despite the challenges posed by variations in pose, clothing, and carrying conditions, which lead to high intra-class variability, our approach utilizes sequence based deep architecture to address these issues. These architectures, with their ability to generalize across different conditions and learn hierarchical and rich feature representations, demonstrate effective representation learning and provide a baseline for comparing the two input modalities. We propose a novel method that utilizes a deep architecture with residual blocks across two modalities to extract direction relevant features. We introduce two different training and testing setting to evaluate our model on CASIA-B dataset and provide ablation study on the effect of triplet loss in training. Experimental results show that our proposed methods achieve impressive Rank-1 accuracy, with an average Rank-1 accuracy of 97.41% on the CASIA-B dataset, an average Rank-1 accuracy of 96.15% and 96.30% for OU-MVLP silhouette and pose dataset respectively.