Background <p>Models that predict gene expression levels from DNA sequence struggle to predict differences between individuals when given their personal genome sequences. These models are generally trained on reference genome sequences, and thus have never observed examples of genetic variation at any locus during training, which may explain their lack of generalizability to personal genome sequences that do contain variation.</p> Results <p>We utilize fine-tuning with personal genomes and matched tissue-specific gene expression values to develop Variformer, a deep sequence-based neural network. Across held-out people, Variformer predicts expression with accuracy that approaches the cis-heritability of most genes and prioritizes genetic variants across the allele frequency spectrum that are enriched for motif disruption and other functional annotations. We highlight how Variformer fails to generalize to unseen genes.</p> Conclusions <p>Our work suggests that fine-tuning with personal genomes corrects previously reported shortcomings of gene expression prediction across unseen individuals, but does not learn a regulatory grammar that generalizes to unseen loci. Fine-tuned deep expression models thus share similar performance and limitations of state-of-the-art linear models, highlighting a gap for the field.</p>