Human brains are natural learning systems which inherently recognise image objects in a hierarchical pattern. Similar association exists among different categories of images which interact while training a deep learning model, leading to misclassification and indistinct features. As semantically related classes form clusters in the embedding space, we can list these classes under a single aggregate class, and make the learning faster by leveraging this knowledge while training a model. In this paper, we introduce a hierarchical architecture that simultaneously learns both broad-class and fine-class embeddings using transformer-based encoders. These encoders are trained alternately, and the hierarchical knowledge is encoded through tensor product between the coarse and fine class representations. Our approach has been able to achieve state-of-the-art performance on benchmark datasets with very less epochs.