In this paper, we propose a multi-agent Deep Reinforcement Learning (DRL) framework to address the Flexible Job Shop Scheduling Problem with Transportation (FJSPT). We decompose the FJSPT into three sub-problems: machine selection, job sequencing, and Automated Guided Vehicle (AGV) scheduling. To model the shop floor layout and machine state features, we employ a graph attention neural network. A policy gradient-based algorithm is used to train the machine selection sub-policy. For job sequencing, we design a transformer-based structure to embed job state features and facilitate action selection, with a Proximal Policy Optimization (PPO) algorithm used to train the job sorting sub-policy. Finally, we develop effective scheduling rules for AGV selection. Experimental results demonstrate that, after individual training, the two agents in our framework outperform handcrafted heuristic dispatching rules in terms of solution quality. Upon completing centralized training, the framework delivers solutions that are qualitatively comparable to state-of-the-art heuristic algorithms while maintaining a short running time. Additionally, the model trained on randomly generated instances performs better on both large-scale and benchmark instances, highlighting the framework's strong generalization capabilities.