We explore the potential role of large language models (LLMs), especially feedforward discriminative transformers trained via next-token prediction, within broader architectures for artificial general intelligence (AGI). We start with practical limitations of current LLMs and analyze their origin in the internal mechanisms. The main question posed is whether LLMs should play a central role in the advancement towards AGI in their current form or their architecture can be incrementally modified to meet AGI requirements, or their role in future AGI architectures can be more peripheral. We come to the conclusion that feedforward LLMs are weak in dealing with large volumes of novel information and to solve genuinely new problems, which cannot be fully fixed by equipping them with external tools. While gradual modifications of LLM architectures may bring them closer to AGI, it will also alter the very nature of LLMs as a sort of effective language reflex or skill. Instead of modifying existing LLMs, they can be used as cached, efficient subsystems for solving frequently encountered language-related tasks. We emphasize that LLMs should be viewed not as complete substrates for general intelligence, but rather as emergent byproducts of learning within an AGI system – a layer of specialized cognitive routines atop a deeper architecture capable of flexible problem-solving across the spectrum from general to domain-specific strategies.