Transformers Compression: A Study of Matrix Decomposition Methods Using Fisher Information

Transformer models have been a breakthrough in Natural Language Processing. However, the performance of these models comes with their enormous size, limiting options for their deployment. Facing this issue, in this paper, we compare different compression techniques, such as low-rank matrix and tensor factorization, for compressing these heavy layers. We focus on Singular Value Decomposition (SVD) and Tensor Train Matrix Decomposition (TTM) and extend previous work [10] by incorporating Fisher information into the TTM, introducing a novel approach which we call FWTTM. In this work, we provide a comprehensive analysis of the performance of the compressed models on different setups and compression levels. We observe a performance increase when using FWTTM compared to other methods on low ranks (high compression rates) for both encoder-only and encoder-decoder models.