<p>Continual Learning seeks to enable artificial intelligence systems to adapt to non-stationary data distributions. We address the challenging problem of Open-Vocabulary Continual Learning (OVCL), where Vision-Language Models (VLMs) must incrementally acquire knowledge from sequential tasks while maintaining generalization capabilities across open-world environments. Unlike prior Multi-domain Task Incremental Learning (MTIL) approaches that rely on explicit task identities during inference, OVCL operates under an ID-agnostic setting in which test samples from diverse tasks appear randomly and must be classified within a unified open-vocabulary label space. To strictly evaluate this setting, we introduce the Generalized OVCL (G-OVCL) benchmark, which incorporates semantically rich unseen samples and interleaved test streams to mirror real-world deployment scenarios. To tackle these challenges, we propose an Isolation-Association-Enhancement (IAE) framework that isolates task knowledge through modular lightweight parameters, dynamically associates relevant knowledge components based on sample-specific visual-semantic relationships, and integrates them via a Bayesian-driven mixture-of-experts strategy for robust, uncertainty-aware adaptation. Extensive experiments on both existing MTIL and our G-OVCL benchmarks demonstrate that our approach clearly outperforms state-of-the-art methods, advancing the development of practical lifelong learning systems in open-world scenarios.</p>