Emotion recognition in conversation (ERC) aims to identify the emotions expressed in utterances by speakers during a conversation. Effective ERC requires modeling both contextual and temporal dependencies. While prior work emphasizes speaker and discourse cues, temporal structure remains underexplored. We propose a Temporal-Aware Attention Network (TAA-Net), which integrates graph-inspired relational encoding into the self-attention mechanism. By incorporating relative temporal positions and time intervals between utterances, TAA-Net enables more precise modeling of temporal dependencies. Experiments on three ERC benchmarks demonstrate the effectiveness of our approach.