The logistic regression estimator is known to inflate the magnitude of its coefficients if the sample size n is small, the dimension p is (moderately) large or the signal-to-noise ratio \(1/\sigma \) is large (probabilities of observing a label are close to 0 or 1). With this in mind, we study the logistic regression estimator with \(p\ll n/\log n\) , assuming Gaussian covariates and labels generated by the Gaussian link function, with a mild optimization constraint on the estimator’s length to ensure existence. We provide finite sample guarantees for its direction, which serves as a classifier, and its Euclidean norm, which is an estimator for the signal-to-noise ratio. We distinguish between two regimes. In the low-noise/small-sample regime ( \(\sigma \lesssim (p\log n)/n\) ), we show that the estimator’s direction (and consequentially the classification error) achieve the rate \((p\log n)/n\) - up to the log term as if the problem was noiseless. In this case, the norm of the estimator is at least of order \(n/(p\log n)\) . If instead \((p\log n)/n\lesssim \sigma \lesssim 1\) , the estimator’s direction achieves the rate \(\sqrt{\sigma p\log n/n}\) , whereas its norm converges to the true norm at the rate \(\sqrt{p\log n/(n\sigma ^3)}\) . As a corollary, the data are not linearly separable with high probability in this regime. In either regime, logistic regression provides a competitive classifier.
