The remarkable parallel computing capabilities of GPUs have led to a notable enhancement in the performance of numerous matrix multiplication operations in deep learning. However, GPUs typically execute computations in an iterative manner and are not optimized for the specific characteristics of deep neural networks (DNNs). Furthermore, current compilers lack the capacity to control thread scheduling, which results in suboptimal utilization of streaming multiprocessors (SMs). These factors contribute to a suboptimal level of computational efficiency. This paper presents a fine-grained scheduling method for SMs that aims to fully leverage the characteristics of deep learning computations and the internal resources of SMs. The method treats SMs as fundamental scheduling units with the objective of achieving more efficient parallelism. We observe that assigning the same task to all SM units is not an optimal approach. Instead, programmers must adopt a more flexible approach to the allocation of computational tasks to each SM unit. This approach positions SMs as scheduling units, thus creating new opportunities for GPU acceleration in deep learning computations, with 1.3x to 3x inference speedup.