This paper examines the problem of disparity between AI systems and human agents in key human characteristics. We point out that this disparity, or gap, despite being recognized, has not been systematically studied. We also argue that without a clearer understanding of this gap, we cannot accurately predict how AI systems will interact with human agents or how they will account for human values and priorities. This paper defines this gap and proposes a methodology for studying it. We also illustrate this conceptual gap with two examples, namely the differences between a synthetic and human consciousness and synthetic agency versus human moral agency. Finally, we emphasize the importance of studying this gap for developing safe AI technology.