Drilling in unconventional shale formations poses significant technical and operational challenges. The effectiveness of the drilling operations in these challenging formations depends on the appropriate selection of the drilling fluid. Oil-based drilling fluids provide excellent performance when drilling these formations. However, the adverse environmental effects of oil-based drilling fluids have restricted its usage globally owing to strict environmental restrictions. Biodiesel-based drilling fluids have become a substitute for traditional oil-based drilling fluids and are favoured for their strong capacity to break down naturally and their little harmful effects. This research work produces biodiesel from the non-edible Calophyllum inophyllum oil via a process of two-step esterification and transesterification. An extensive experimental study was conducted to analyse the rheological behaviour, filtration characteristics, and emulsion stability of the formulated drilling fluid under both normal and reservoir conditions. In addition, shale inhibition tests were also performed to examine the effect of the formulated drilling fluid on the swelling properties of the clay-rich shale formation under high pressure and high temperature conditions, with an exposure duration of 24 h. The drilling fluid’s formulation improves rheological properties, increases emulsion stability and lowers filtration loss following API standards. The drilling fluid demonstrated outstanding performance with a maximum clay swelling of 5.4% at high pressure and high temperature conditions. The outcomes demonstrate that the formulated biodiesel-based drilling-fluid has the potential to substitute oil-based drilling fluid as an appropriate technical and environmental alternative for drilling unconventional shale formations.