This study evaluates the color perception capabilities in vision language models (VLM) by studying the impact of adopting three different visual feature extraction methods and the Visual Instruction Tuning method on color understanding. We created a customized Visual Question Answering (VQA) dataset of simple geometric shapes in various colors against a colored background, paired with diverse question types ranging from basic color identification to relational reasoning. Twelve VLMs are used in our evaluation and divided into two groups: the first group contains VLP models that employ three distinct visual feature extraction approaches: region-based (VisualBERT-FRCNN), grid-based (VisualBERT-ResNet), and patch-based (ViLT, BLIP). The second group includes recent state-of-the-art Multi-modal Large Language Models (MLLMs) that adopt visual instruction tuning. The performance of the two groups varies across different color perception tasks, where the first group failed to identify the color categorization task and showed an inability to identify the cyan and magenta colors. The second group, including models like BLIP-2, InstructBLIP, LLaVA-1.5, Gemma 3, Qwen2.5-VL, GPT-4.1, GPT-4.1 mini, and GPT-4.1 nano, show varying performances across the color perception tasks. BLIP-2, InstructBLIP, and LLaVA-1.5 were unable to identify the cyan and magenta colors, but can identify the color category. The performance of GPT-4.1 outperforms all the adopted models, including the Qwen2.5-VL models, which shows competitive performance despite being open source.