Artificial intelligence-driven techniques for pose estimation and behavior analysis represent a significant advancement in the elucidation of precise movement trajectories and behavioral components in freely moving animals. However, comprehending the underlying temporal dynamics of these trajectories remains challenging due to the dearth of effective analytical instruments. In order to decipher the nuanced body language inherent in behavioral dynamics at a sequential level, we introduce BL-BERT, a computational framework rooted in Bidirectional Encoder Representation from Transformers (BERT). This framework discerns stereotypical behavior sequences exhibited by freely moving mice, elucidating behavioral dynamics in a linguistically comprehensible manner. BL-BERT discerns salient behavior sequences from input behavior modules, as evidenced by its performance on a custom dataset of interactions among free-moving mice. Diverging from conventional Markov models, BL-BERT unfolds the recurrent structure of behavior sequences, rendering it more interpretable. BL-BERT offers a novel way to apprehend the hierarchical organization of intricate animal behaviors, with promising prospects for widespread applicability across various behavioral paradigms.