Short-pulse, high-intensity lasers offer a compact alternative to linear accelerators for driving high-flux neutron sources. One method to produce laser-driven neutrons is in a so-called “pitcher-catcher” setup, wherein deuterons and protons are accelerated into a converter material to produce neutrons. In this study, deuteron and proton spectra, along with angular ion divergence measured during experiments conducted at Omega EP (laser parameters: 500 J, 0.7 PS, 7 × 1019 W/cm2), are used to optimize converter material and geometry for neutron radiography. The optimization process employs 3520 Monte Carlo N-Particle (MCNP®) simulation cases to build an Artificial Neural Network (ANN) surrogate model of neutron yield characteristics based on incident particle type (proton or deuteron), converter material (LiF or Be), and converter dimensions (thickness and radius of cylinders or cones). Using the surrogate model with the multi-objective genetic optimization algorithm we identify the Pareto front producing the optimal parameter combinations. The main parameters of interest are neutron yield and directionality, with an additional goal of minimizing converter volume. Relative errors between the Monte Carlo validation results and the surrogate model optimization results for each modeled objective range 0.1–2% in terms of mean absolute relative error. The resulting converter designs generate beams of up to 2.44 × 1010 neutrons per pulse, exhibiting the desired beam-like fast neutron angular emission profiles without requiring collimation. This successful demonstration of multi-objective optimization for source development can be extended from fast neutron beam development to thermal neutron source development.