Since the advent of the Transformer, there have been numerous significant studies in the language domain, and it has also impacted the visual domain, where many visual studies integrating Transformers have achieved remarkable performance. A classic example is the Vision Transformer (ViT). ViT utilizes a global self-attention mechanism, allowing any block in the image to interactively compute with other blocks. However, in many images, non-critical information occupies most of the blocks, and in reality, excluding these blocks from computation has minimal impact on decision-making outcomes. Therefore, we developed a method based on a patch-based approach, utilizing global information through a grid table, abbreviated as GT-RAM. Through training methods from reinforcement learning, GT-RAM only collects samples from certain blocks for image classification. In experiments using the Mnist and Fashion datasets, compared with different computational approaches like CNN and ViT, GT-RAM’s accuracy exceeded the former two methods, demonstrating that even with only key blocks sampled, the model can perform well. The sampling efficiency also surpassed similar architectures such as RAM and DT-RAM. This research confirms the effectiveness of the patch-based sampling method, and the straightforward approach of GT-RAM, which avoids redundant sampling, contributes to future studies on cooperative sampling among multiple agents.