<p>We have witnessed the emergence of superhuman intelligence thanks to the fast development of large language models (LLMs) and multimodal language models. As the application of such superhuman models becomes increasingly popular, a critical question arises: how can we ensure they still remain safe, reliable, and aligned well with human values encompassing moral values, Schwartz’s Values, ethics, and many more? In this position paper, we discuss the concept of superalignment from a learning perspective to answer this question by outlining the learning paradigm shift from large-scale pretraining and supervised fine-tuning, to alignment training. We define superalignment as designing effective and efficient alignment algorithms to learn from noisy-labeled data (point-wise samples or pair-wise preference data) in a scalable way when the task is very complex for human experts to annotate and when the model is stronger than human experts. We highlight some key research problems in superalignment, namely, weak-to-strong generalization, scalable oversight, and evaluation. We then present a conceptual framework for superalignment, which comprises three modules: an attacker which generates the adversary queries trying to expose the weaknesses of a learner model, a learner which refines itself by learning from scalable feedbacks generated by a critic model with minimal human experts, and a critic which generates critics or explanations for a given query-response pair, with a target of improving the learner by criticizing. We discuss some important research problems in each component of this framework and highlight some interesting research ideas that are closely related to our proposed framework, for instance, self-alignment, self-play, self-refinement, and more. Last, we highlight some future research directions for superalignment, including the identification of new emergent risks and multi-dimensional alignment.</p>