Threatening rumors diffuse more quickly and widely than ever with the popularity of social media. Therefore, it is crucial to identify a rumor as early as possible to stop its spreading and reduce the potential damages. Mainstream social media rumor detection methods utilize the content or propagation information but ignore the global temporal information, i.e., timestamps of published posts, therefore fail in capturing the variant attentions of the interactive pattern on different time intervals and determining a dynamical time point to debunk for each individual rumor. Meanwhile, the absence of global temporal information cannot provide a whole picture of the rumor propagation structure and will reduce the robustness of the model. For this purpose, we present a novel Temporal-aware Graph Attention Network (TGAN), for early rumor detection. In TGAN, the cascade of a rumor can be represented with a graph neural network, where each node denotes a post while the edge denotes the interactive pattern among posts. The global temporal information is integrated in TGAN, (1) to capture neighborhood attention and interactive attention to represent the structural information; (2) to learn the potential influence of a post and estimate the burst time of a rumor for early rumor detection. More importantly, the rumor can be identified earlier before the burst time, while maintaining a promising accuracy.