Annotations were traditionally a way to add comments and extensions to manuscripts, which developed into the footnote as a formalised system for second layers in texts. With the development for 3D modelling as a means to digitise physical objects, as well as to make born digital models, annotation was also extended to such objects. Based on this wider view on annotations, a truly integrated semantic web for cultural heritage can be developed, based on contextual modelling of memory twins. This paper describes this development and discuss where it might go from here, seeing annotations as mechanisms for the integration of complex objects across media borders.