The Tensor Processing Unit (TPU) emerges as a specialized hardware accelerator tailored for the demanding computations inherent in deep learning tasks. Initially developed by Google, TPUs have gained widespread attention due to their remarkable efficiency and performance in training and deploying neural networks. The Matrix Multiplication Unit (MMU) of the Tensor Processing Unit (TPU) is the computation-intensive unit. The improvement in the MMU thus results in significant enhancement of the system by increasing the computational speed and decreasing the energy consumption and area required. The novel MMU is designed to achieve parallelism and efficient use of memory. The proposed design is implemented in TSMC 180nm CMOS Technology. It is observed that the proposed design consumes 57.07 % less power, 4.387% less cell count, and is 76.17% faster compared with the reference design.