End-to-end mapping from human to robot hand poses offers a compelling, non-invasive approach for translating visual hand data into robot joint commands. However, existing methods rely heavily on labor-intensive, hand-structure-specific paired datasets, hindering scalability and cross-device adaptability. To address these limitations, we propose End-to End Partial Supervision Mapping (EPSM), a novel framework that decouples pose information from structure-specific information through a shared latent grasp feature space. EPSM employs a unified encoder to extract low-dimensional grasp representations from depth images, while separate decoders reconstruct robot joint angles tailored to specific hand structures. By sharing encoder weights across models and isolating decoder training to individual robotic hands, EPSM enables cross-structure mapping via decoder swapping–without requiring any human-robot paired training data across different configurations. We demonstrate the effectiveness of EPSM by mapping human hand poses to three robotic hands with varying finger counts (three, four, and five), showcasing its potential as a generalizable and scalable solution for human-robot hand interaction.