Gaussian Process-based Bayesian Optimization (BO) has proven effective in Hyperparameter Optimization and Neural Architecture Search play a crucial for optimizing continuous search spaces. However, its performance is hindered by categorical variables inherent in these tasks, as standard Gaussian Process explicitly assume a continuous input space. As a result, effectively handling categorical variables remains a significant challenge for Gaussian Process-based BO. To address this issue, we propose Adaptive Linear Mapping Bayesian Optimization (ALMBO), a multi-fidelity BO method that leverages the Adaptive Linear Mapping Gaussian Process (ALMGP) module to process categorical variables. ALMGP adaptively captures correlations both between categorical variables and within their categories, enhancing the Gaussian process modeling. Additionally, we propose the DG-EI acquisition function, which balances exploration and exploitation to further enhance ALMGP’s effectiveness. Furthermore, by integrating the proposed ALMGP module and DG-EI acquisition function into the existing multi-fidelity framework FastBO, we establish a multi-fidelity optimization architecture that effectively handles categorical variables, ensuring better resource utilization. Experimental results demonstrate that ALMBO effectively balances resource utilization and performance, ultimately converging to hyperparameter configurations that are closer to the global optimum, showcasing its strong optimization capabilities.