Fast and accurate real-time 3D hand-object pose estimation is generally recognized as a challenging scenario, specifically in the absence of depth-sensing cameras, high-performance computing and storage hardware. We propose a new lightweight, fast and accurate model for 3D hand object pose estimation, which takes continuous RGB frame sequences as input and introduces the use of temporal graph convolutional networks to optimize estimation accuracy. Compared to the state-of-the-art RGB-based methods, our proposed model achieves real-time performance with reduced computational resource requirements, a smaller model size, and superior accuracy compared to previous lightweight models (and comparable accuracy to larger, more computationally intensive state-of-the-art models). The performance of our method has been verified on the First Person Hand Action (FPHA) and HO-3D benchmarks.