Human-object interaction (HOI) detection aims to localize human-object pairs and recognize their interaction categories, playing a critical role in high-level visual understanding. While recent methods leverage vision-language models (VLMs) for improved generalization, most approaches focus predominantly on visual knowledge, underutilizing the rich semantic potential of language features. In this paper, we propose VLHOI, a novel VLM-based HOI detection framework that jointly transfers visual and linguistic knowledge from VLMs to enhance zero-shot and regular HOI detection. To generate image-aligned textual semantics, we employ an image captioning model and further refine descriptions via a large language model (LLM). These enriched descriptions are encoded with CLIP and progressively fused with visual and interaction features through deep knowledge fusion modules. Extensive experiments on the HICO-DET dataset show that VLHOI achieves state-of-the-art performance in both regular and zero-shot settings, with significant improvements in unseen categories. Our results validate the importance of jointly leveraging vision and language information for HOI understanding.