<p>Modular crossed product algebras have recently assumed an important role in perturbative quantum gravity as they lead to an <i>intrinsic</i> regularization of entanglement entropies by introducing quantum reference frames (QRFs) in place of explicit regulators. This is achieved by imposing certain boost constraints on gravitons, QRFs and other fields. Here, we revisit the question of how these constraints should be understood through the lens of perturbation theory and particularly the study of linearization (in)stabilities, exploring when linearized solutions can be integrated to exact ones. Our aim is to provide some clarity about the status of justification, under various conditions, for imposing such constraints on the linearized theory in the <i>G</i>_<i>N</i> → 0 limit as they turn out to be of second-order. While for spatially closed spacetimes there is an essentially unambiguous justification, in the presence of boundaries or the absence of isometries this depends on whether one is also interested in second-order observables. Linearization (in)stabilities occur in any gauge-covariant field theory with non-linear equations and to address this in a unified framework, we translate the subject from the usual canonical formulation into a systematic covariant phase space language. This overcomes theory-specific arguments, exhibiting the universal structure behind (in)stabilities, and permits us to cover arbitrary generally covariant theories. We comment on the relation to modular flow and illustrate our findings in several gravity and gauge theory examples.</p>