A novel topology optimization framework is established for orthotropic multi-material structures based on Neural Network (NN) and element free Galerkin method (EFGM). The global coordinates and relative density of EFG nodes in orthotropic multi-material structures are selected as inputs and outputs for a fully connected feedforward neural network (FNN), respectively. Sensitivity analysis is automatically performed using backpropagation technology of the proposed topology optimization framework, and the Adam optimizer is used to adjust network parameters. The computational efficiency of EFGM(Adam_FNN) algorithm and traditional algorithm is compared. The effects of network depth, neural network learning rate, and the off-angle of anisotropic materials on topology results are studied using several numerical cases. The results indicate that the proposed framework can effectively reduce the computational cost of orthotropic multi-material structure topology optimization while maintaining excellent numerical accuracy. A lower network depth will result in intermediate density in topology configuration, the quantity of hidden layers is suggested to be more than 4. A larger learning rate may lead to difficulty in convergence. The compliance of the optimal orthotropic multi-material topology can be reduced by adjusting the off-angle, the off-angle of multi-material structure is suggested to be 0°–30° or 120°–150°.