Graph Representation Learning (GRL) has emerged as a thriving field within machine learning, leveraging Graph Neural Networks (GNNs) to extract meaningful representations of graphs. The typical evaluation framework however relies heavily on a limited set of empirical datasets which appear, each and as a whole, to cover a rather narrow segment of the potential graph space. We propose a novel synthetic benchmarking framework capable of spanning extensive portions of the space of possible graphs, both in terms of basic size and sophisticated topological properties. This helps us demonstrate that several state-of-the-art models exhibit task-specific performance and may not generalize well. For some regions of the space, naive graph representation methods such as degree distributions may perform even better. Our contribution makes it possible to discern regions where computationally intensive GRL methods offer substantial value and, more broadly, emphasizes the need for a broad, tunable benchmark covering an array of graph properties. We encourage the integration of such synthetic benchmarks to complement the empirical appraisal of GRL models.