Nanbanjin—‘southern barbarians’—was the term used by the Japanese to refer to the first Europeans to arrive in Japan, in 1543. Initially, these men—with their strange appearances and habits, unfamiliar languages, and new and alien religion—provoked curiosity. By 1639, however, perceptions had changed, and the nanbanjin had come to be associated with deviant behaviour and unconventional conduct, similarly to the kabukimono—local people living at the margins of society, who often borrowed elements of nanbanjin dress and behaviour as part of their aesthetic of subversion. The nanbanjin were no longer welcome in Japan, and mistrust prevailed over the initial curiosity. This chapter analyzes the portrayal of nanbanjin in Japanese written accounts and images in the period, aiming to shed light on the evolution of their identification through a complex translation process within the context of early modern transnational relations.