Prior research on perception of a vocalization as speech versus song has focused on how stimulus characteristics are mapped onto the boundary between these domains. It remains an open question, therefore, whether the perception of a stimulus as speech versus song can shift based on the context surrounding a stimulus. In Experiment 1 participants rated stimuli which varied along a continuum from speech to song. These ambiguous stimuli were preceded by adaptor stimuli which were unambiguous song or speech. Stimuli preceded by a speech adaptor were rated as more song-like, while stimuli preceded by a song adaptor were rated as more speech-like. In Experiment 2, data from a previous study on transformation from speech to song with repetition were re-analyzed to examine contextual effects. After hearing a stimulus which transformed into song, participants were more likely to rate the next stimulus as sounding more speech-like compared to after hearing a stimulus which continued to sound like speech when repeated. These findings show that when listeners decide whether a stimulus is speech or song, they do not only use acoustic information but also take into account contextual information across multiple time scales, from a few seconds (Experiment 1) to around 15 s (Experiment 2). Thus, the perceptual boundary between speech and song is not fixed.