<p>The pathologists grade cancers by identifying the morphology of tumor areas, whereas this manual implementation is time-consuming and expensive. The most recent methods deploy deep-learning architectures (like CNNs or Transformers) to conduct pathology image segmentation automatically. However, these implementations are not satisfactory for this modality where the pathological medical images show huger domain gap and more complex morphological characteristics. This paper contributes a novel approach based on the Diffusion Model to accurately segment the lesion areas for pathology images. Specifically, we introduce a scale-wise way of hierarchical guidance production to yield multiscale features to capture informative semantics of the pathology images. Moreover, in a scheme-wise manner, we make use of the hierarchical guidance to guide conditional Diffusion Model to progressively and accurately decode the final mask. Our Hierarchy Aggregated Diffusion (dubbed as HADiff) performs well for capturing the morphology of tumor regions for both cell level and tissue level, overtaking the state-of-the-art methods by a large margin. Extensive experiments and strong results on three public datasets suggest that our approach is of robustness and generalization for pathology image segmentation. Codes and weights will be released at <a href="https://github.com/ge-xing/HADiff">https://github.com/ge-xing/HADiff</a>.</p>