Analyzing RNA-Seq Data in Complex Study Designs

Recently, RNA-seq experiments have become a routine technique in studying the transcriptomic regulations in various biomedical problems. The experimental design has extended from simple two-group comparisons with only a few replicates to complex study designs involving multiple treatment groups and repeated measurements. We present that even when systemic biases are removed by normalization procedures, there are systemic heteroscedasticity that makes observations from some samples noisier than from others. We develop a novel method that accurately estimates the heteroscedasticity. Accounting for it in weighted regression leads to increased true discovery in differential expression (DE) detection in both independent and clustered (longitudinal) samples. We also present a new measure of effect size in DE in longitudinal studies, when the treatment effect is expected to last over a period and peak at different time for different genes. We illustrate the advantage of the proposed method over existing approaches through extensive simulations and demonstrate the application in a real data from a longitudinal study on vaccine efficacy for Dengue fever where a two-dose heterologous prime-boost vaccination regime is utilized.