Work occupies a central place as part of the symbolic and material order of recognition in modern societies. However, the very social definition of work as a normative concept includes in itself a disdain toward all those who do not work or are considered as not doing so properly. The chapter explores how the argument of work is used to disrespect different social groups, reflecting upon the possibility that the very same normative idea of work as it is institutionalized in wage labor is a form of the modern justification of discrimination toward the worst off. The article analyses racist discourses formations about the “lazy” gypsy population in Spain and basic structures of the antisemitic argument against the “greedy” Jews. It is not through the “other” but rather through a theoretical understanding of the processes of othering, racism, antigypsyism and antisemitism that we can learn about the failures of the forms of institutionalization of recognition in today’s society. This approach serves as an example of immanent critique in the classical sense, by measuring the institution of work according to its own normative standard and showing that policies aiming at a generalization of inclusion in this form of work have an immanent tendency to fail. The article shows how the social notion of work reproduces images of others as parasitic forces within the otherwise brave new world of modern work.