This work investigates recently proposed robust versions of neural networks and particularly their regularized versions, which are recommendable for regression models with a large number of parameters. Common types of regularization have the form of penalization (shrinkage) applied to the loss function. Highly robust multilayer perceptrons based on ranks of residuals are considered here. One aim is to formulate their interpretation within the framework of Bayesian inference; attention is paid to a specific situation with the prior information in the form of additional measurements. Another aim is to propose a procedure for outlier detection, which is based on a sequential instance selection. The task corresponds to tuning the breakdown point. Numerical experiments illustrate the procedure and also reveal the benefits of the regularization.