Autoregressive models have demonstrated considerable efficacy in sequence modeling tasks, including the recognition of formal languages. These models predict the next symbol in a sequence, making them particularly well-suited for tasks that require an understanding of structured patterns. However, their performance in recognizing formal languages can be compromised by statistical biases in the training data, such as imbalanced symbol distributions, overrepresented linguistic structures, or insufficient examples of certain formal patterns. These biases can hinder generalization and degrade recognition accuracy. Although autoregressive models have the potential to approximate complex language structures, the presence of statistical biases may impair their ability to fully capture the intricacies of formal languages, especially those with strict syntactic rules or irregular structures. This study investigates how the inductive biases inherent in current autoregressive modeling approaches interact with statistical biases and examines their impact on the models’ ability to learn and recognize formal languages. Specifically, we assess the out-of-distribution (OOD) generalization performance of various autoregressive architectures trained on several formal languages, with sequence lengths sampled from distributions that (a) simulate distribution shifts, (b) represent a bias towards certain sequence lengths in the train data. Our results reveal that training on biased sequence lengths adversely affects OOD performance. Furthermore, we demonstrate that some architectures, due to their inductive biases, exhibit a preference for particular tasks, and that exposing them to specific sequence length distributions can improve OOD performance.