Precise estimation of speaker physical attributes such as gender, age, weight, and height, from speech signals is a challenging problem. This research proposes a unique approach that utilizes efficient classifiers and regressors that make use of optimized and discriminative set of features that are carefully chosen from speech signals, to address this issue by leveraging acoustic features and machine learning techniques. We investigate the relationship between physical characteristics and vocal tract parameters, utilizing features like MFCCs, pitch, intensity, jitter, and shimmer. To classify gender and estimate the continuous values of height, age, and weight, we use SVM, kNN, and SVR algorithms. Experiments conducted using the TIMIT dataset and a custom made dataset illustrate the capability of the method proposed for applications in various fields, including forensics, healthcare, and human–computer interaction.