The task of hierarchical multi-label classification (HMLC) involves predicting structured outputs, where classes are organized in a hierarchy and instances can be associated with multiple classes simultaneously. Despite the popularity of HMLC in the machine learning field, there is a lack of HMLC datasets and methods in the remote sensing (RS) domain. Therefore, it is still not clear whether considering the hierarchical relationships among classes of remote sensing images (RSI) is advantageous or not. The CORINE Land Cover (CLC) nomenclature, which provides detailed information about the land cover classes at multiple levels of the hierarchy, makes it possible to construct label hierarchies for RSI datasets and explore the relevance of HMLC in the RS domain. In this work, based on the CLC nomenclature, we assemble label hierarchies for the BigEarthNet and UCM datasets, constructing HMLC version of these datasets, which we release to the public as the first HMLC RSI benchmarks. Furthermore, we show that incorporating hierarchical information about the classes brings advantage in predictive performance over the prevalent multi-label classification setting.