We consider the supervised clustering problem under the two-component anisotropic Gaussian mixture model in high dimensions and in the non-asymptotic setting. We characterize precisely the risk of l2 regularized supervised least squares classifiers. We deduce the fact that the interpolating solution may outperform regularized classifiers, under mild assumptions on the covariance structure of the noise. Our analysis also shows that overfitting can be robust to corruption in the covariance of the noise when the signal is aligned with the "clean'' part of the covariance, for the properly defined notion of alignment. To the best of our knowledge, this peculiar phenomenon has not yet been investigated in the rapidly growing literature related to interpolation. We conclude that overfitting is not only benign but can also be robust.
This is joint work with Stanislav Minsker and Yiqiu Shen.