Emotion Recognition in Conversation (ERC) is the task of identifying the emotional label of each utterance in conversation. Current studies mainly focus on better modeling the dialogue context, while the emotional dynamics of conversation are ignored. The dynamics of emotions in conversation are very obvious. On the one hand, emotions have a certain inertia. To some extent, the speaker easily keeps his or her emotion the same. So the former emotion can contribute much to recognizing the latter. On the other hand, emotion can be influenced by what is being said and who is speaking. Along with the topic change and the speaker’s state, so do the emotions expressed in dialogue. In this paper, we propose a encoder-decoder framework to explicitly incorporate the emotional dynamics in dialogue. In the encoder, we exploit GRU to track the emotional flow of individual speakers and entire conversations. In the decoder, we introduce Emotion Shift Detection auxiliary task to evaluate emotion shift probability between adjacent utterances in the dialogue level and speaker level. When identifying the emotion of the target utterance, the emotion of the previous utterance is considered as well as the predicted emotion shift probability is employed to guide emotion recognition. Besides, we use several popular conversational context encoders to show our proposed approach’s high scalability and flexibility. Experiments on two benchmark datasets also indicate the great effectiveness of our approach.