Compound channels, commonly found in natural river systems, typically consist of a main channel alongside one or two floodplains. In prismatic compound channels, water levels remain uniform, whereas non-prismatic ones exhibit a more complex and non-uniform flow pattern. The interaction between the main channel and floodplain generates shear stresses due to momentum transfer, but predicting their distribution along the boundary is challenging as it depends on various factors like velocity distribution, cross-section shape, and boundary roughness. This study aims to develop a model to predict shear stress in non-prismatic compound channels using two machine learning (ML) techniques: M5 tree and extreme gradient boosting (XGBoost). It considers geometric, flow, and roughness parameters such as width ratio, relative flow depth, flow aspect ratio, and bed slope. M5 tree offers transparent formulas that enhance understanding, while XGBoost, known for handling structured data well, is a powerful gradient-boosting algorithm. Both models utilize correlated physical channel characteristic variables to predict shear stress. The predictions from both M5 tree and XGBoost models demonstrate satisfactory performance, with coefficients of determination (R2) exceeding 0.85 and mean absolute percentage errors (MAPE) below 12% for both training and testing datasets. However, the XGBoost model generally outperforms the M5 tree model in predicting shear stress across different sections of non-prismatic compound channels.