Table Structure Recognition (TSR) aims to extract two-part information from a table image: a 2D structured language sequence and a bounding box sequence. Image-to-text (i2t) methods have received more attention among recent TSR methods. However, recent i2t methods use (1) a dual-decoder framework for the two-part output, which is complex and hard to design, and (2) the vanilla attention layer for predicting 2D structured sequences, which is inappropriate because the vanilla attention is designed for 1D sequences. To address these problems, (1) we are the first to propose a novel encoder-decoder framework based on i2t methods, which discards the old dual-decoder framework, making it more uniform and easier to design. Our encoder-decoder architecture is composed of three modules: Multi-Modal Encoder, Multi-Modal Mid-Block, and Multi-Modal Decoder. (2) We also propose a novel 2D attention layer, which explicitly models the 2D features of the structured language. Finally, we test our method on public datasets and achieve significant improvements in visual prediction and comparable results in text prediction.