Protein tertiary structures capture the complex three-dimensional (3D) arrangements of the constituent atoms that define how proteins function. Different strategies have been applied in the form of representation learning to interpret these high-dimensional structure spaces. Factorization-based techniques, such as matrix and tensor factorization, have also been proven to be an effective set of approaches due to their utility in capturing latent organizations from complicated structural spaces and have already been applied in the context of tasks like tracking conformational changes across structure ensembles of biomolecules, identifying the biologically-active tertiary structure(s) from the given computed protein models, analyzing biomolecular dynamics simulations, etc. This paper exhibits a comprehensive review of factorization-driven methods employed to learn representations of tertiary structures for the protein structure prediction task. It covers how protein structures are encoded, represented, analyzed, and evaluated via different factorization-based frameworks. Furthermore, it outlines major open challenges and offers prospective research directions.