In-air handwriting is a new type of non-contact information input method. Current research primarily uses LSTM to model in-air handwritten text line, but LSTM is limited in its ability to fully capture the relevant features due to the inherent continuity and complexity of this task. To investigate this, we first apply the more powerful vanilla transformer architecture but find its performance is modest. Further analysis reveals that this is caused by an attention assignment issue, which refers to the vanilla transformer tends to overly focus on short segments of the writing trajectory, which hinders accurate character recognition. To solve that, we propose an Attention-Modulated Transformer (AMFormer). Specifically, for each transformer block, its attention value is modulated based on the values of previous blocks, which make the attention of different transformer blocks consistent, preventing the model to overly focus on short segments. Extensive experiments demonstrate that the proposed AMFormer respectively outperforms the state-of-the-art LSTM-based counterparts on in-air handwriting recognition benchmarks.