Antioxidants play a vital role in preserving the quality of lipid-based food products by delaying oil oxidation. Many research studies have revealed that synthetic antioxidants can pose uncountable risks to human health. Therefore, a search for natural alternatives is a pressing matter of the moment. Virgin olive oil is one of the great sources of antioxidants, but its color has made its practical utilization as an efficient natural antioxidant in lipid-based food product becomes more difficult due to its dark yellow color. Thus, decolorization is necessary as it will significantly affect the final appearance of the food products. The objectives of this study are to decolorize virgin olive oil using the adsorption method and to optimize the decolorizing parameters. In this study, the decolorization performance of activated carbon, clinoptilolite, multi-walled carbon nanotubes, kaolin, and sodium-based bentonite clay are investigated. It is found that kaolin has the best decolorization performance with an antioxidant percentage reduction of 13.32% and an average rating of sensory evaluation 1.8 out of 5. The decolorization parameters, which are temperature, ratio of virgin olive oil to adsorbents, and contact time are further optimized by using the Taguchi method. After experimenting, the optimal decolorization parameters are a 3:1 oil-to-adsorbents ratio, 5 days, and 50 °C, giving an antioxidant percentage reduction of 16.10% and a color average rating of sensory evaluation of 1.63 out of 5.