The numerical simulation of brittle fracture phenomena using the phase-field method is addressed in this contribution. A combined FE-SBFE approach is proposed, which facilitates the use of hierarchical meshes. Thus, rapid element size transition near cracks is achieved. Here, polygon shape functions based on the scaled boundary finite element method (SBFEM) are employed in square elements with hanging nodes, whereas classical finite element shape functions are used in four-noded elements. Thus, the more expensive numerical integration of the polygon shape functions is avoided in large regions of the mesh. An adaptive mesh refinement strategy is summarized and applied to three classical benchmark examples. A significant reduction of computational cost is achieved compared to a pure SBFE-based approach.