GAN inversion is essential for face editing, yet most existing methods emphasize identity preservation while neglecting fine-grained attribute consistency. We present a two-step framework that jointly enforces identity and attribute fidelity, yielding reconstructions that are both visually accurate and semantically consistent. Our approach integrates the multitask transformer SwinFace into encoder training (pSp and e4e) and per-image refinement, replacing ArcFace and leveraging attribute embeddings. To assess reconstruction quality, we introduce evaluation protocols tailored to identity consistency (FIC), attribute consistency (FAC), and face image quality (FIQA), in addition to standard perceptual metrics. Experiments on CelebA-HQ, FRL, and FERET demonstrate that our method reduces both identity and attribute change while improving FIQA, without compromising perceptual similarity. These results confirm that explicitly modeling attributes alongside identity is crucial for a faithful and generalizable GAN inversion.