Reinforcing polymer matrix with nanofillers is known to be an effective method to improve the mechanical properties of a nanocomposite. However, nanoparticles such as carbon nanotube and graphene tend to agglomerate inside a polymer nanocomposite (PNC). Agglomeration of nanoparticles can lead to a zone of stress concentration. This stress concentration can further cause the failure of PNC due to matrix cracking. This study adopts a hierarchical multiscale method to predict the effect of nanoparticle agglomeration inside a PNC. Molecular dynamics (MD) simulations are performed at the nanoscale. The information from these MD simulations is used as an input to the finite element analysis. The adopted multiscale technique also predicts the effective properties of the PNC at a low volume fraction comparable to experiments. Obtaining such low volume fractions using MD simulations is impossible due to the computational cost involved. The results from the simulations indicate that the stress concentration increases as the nanoparticles are close to each other. Also, the stress across the cross-section of the polymer is observed to be highest in the agglomerated region. The effective properties predicted in this study are in good agreement with the values reported in experiments.