Most weakly supervised semantic segmentation (WSSS) methods use class activation map (CAM) to generate pseudo labels. However, due to the image-level supervision, the CAM obtained from the classification network can only activate the most discriminative regions instead of the whole object. A few attempts have studied class-agnostic CAM generation for this issue. Nonetheless, they may lose class semantics, thus degrading semantic segmentation performance. In this paper, we design a two-branch CAM generation network to solve the previous challenges, including a class-agnostic and a classification subnetwork. To balance these two contradictory subnetworks, we introduce a gradient-balanced mechanism to control the gradient propagation during the backpropagation stage. In addition, we employ the part-whole relational property of Capsule Networks (CapsNets) in the classification branch, which is intended to reduce the over-emphasis on class-specific local details in foreground regions. The CAM generated from the class-agnostic branch produces a background cue map for further semantic segmentation. Extensive experiments on the PASCAL VOC 2012 and MS COCO 2014 datasets demonstrate the superiority of the proposed approach.