This study presents a novel approach aimed at enhancing the Perceptron Algorithm by incorporating Bayesian Optimisation based on the cross-validation framework. We bring in Bayesian Optimisation techniques, leveraging the upper bound of a credible interval as an acquisition function to guide the search for optimal parameters. Through extensive simulations on artificial datasets, we demonstrate that this Bayesian-optimised Perceptron consistently outperforms the original algorithm across various dimensions. This work highlights the significance of robust statistical evaluation and Bayesian Optimisation in improving the performance and generalisation ability of even simple machine learning models like the Perceptron. This research could also lay the groundwork for future studies on applying Bayesian Optimisation to enhance the performance of Multilayer Perceptron Networks.