Arrival/Travel times for public transit exhibit variability due to factors like seasonality, traffic signals, travel demand fluctuation etc. The developing world in particular is plagued by additional factors like lack of lane discipline, excess vehicles, diverse modes of transport, unreliable schedules etc. This renders the bus arrival time prediction (BATP) to be a challenging problem especially in the developing world. A model based on Encoder-Decoder (ED) OR Seq2Seq recurrent neural networks (RNNs) is proposed for BATP. The model incorporates spatio-temporal (ST) correlations in a unique fashion. Existing ED approaches for BATP blindly map time to the sequential aspect of ED, while ignoring crucial data characteristics and making restrictive modelling assumptions. We exploit the geometry of the dynamic real-time BATP problem to enable a novel fit with the ED structure, distinct from existing ED approaches. Motivated from accurately modelling past congestion influences from downstream sections, we propose a bidirectional layer at the decoder. Effectiveness of the proposed architecture is demonstrated on real data.