Vision-language models have undergone significant development in recent years, integrating computer vision and natural language processing technologies to analyze and generate text information related to images. These models are extensively applied in various scenarios, including image captioning, interactive question answering and automatic tagging, etc. However, training these models requires vast amounts of data and computational resources, thereby increasing their value as critical assets. Consequently, the security concerns associated with vision-language models are non-negligible, particularly regarding model stealing attacks. In such attacks, perpetrators can acquire a surrogate model with similar capabilities to the target model. While model extraction attacks have been deeply studied, they focus on single modal models and cannot be directly transferred to multi-modal models. To this end, we propose a novel model extraction attack against vision-language models, called VLME. Specifically, VLME understands the relationship between different input and output data via knowledge distillation patterns. By effectively mapping information between these disparate modalities, we have developed a novel loss function tailored for training the surrogate model. Leveraging insights gained from the information mapping process, we successfully conduct model stealing attacks on different distribution datasets and various model architectures through image-to-text and text-to-image APIs. The results demonstrate that VLME has better performance than existing work and owns the advantages such as being query-efficient, API-independent, data-independent, and architecture-independent. This significantly expands the applicability of our attacks. Our research underscores the privacy and security concerns of vision-language models and aims to engage relevant works in addressing these challenges.