This work provides a new approach for modeling of inhomogeneous viscoelastic deformable objects. The approach is validated on a dataset collected at multiple locations of three different inhomogeneous deformable objects. The dataset consists of position and force measurements corresponding to single finger normal interaction. The approach, first, employs the principles of feature-based learning and perceptual adaptive sampling mechanisms to reduce the dataset. Then, a single random forest-fractional derivative (RF-FD) based data-driven model is trained on the reduced dataset to estimate a non-parametric relation between position and force samples for each deformable object. Thus, the proposed approach requires just one trained model to predict interactions at unknown locations of the object with good accuracy, unlike the existing clustering based solution in the literature where one model is trained for each cluster. Our results demonstrate that the proposed approach provides a better prediction accuracy in estimating the responses on inhomogeneous objects as compared to the existing solution in the literature in terms of the relative root mean square error (less than 0.15) and maximum error (less than 0.75 N).