Bagging regularized M-estimators: Precise asymptotics and cross-validation

Ensemble methods improve model stability and generalization by averaging predictions from multiple base learners. A canonical ensemble method is bootstrap aggregating aka bagging, which trains models on resampled datasets and averages their outputs, and its cousins, subagging (subsampled bootstrap aggregating), which uses subsampled datasets. Beyond computational benefits, subagging can improve generalization, especially in overparameterized regimes near interpolation thresholds (e.g., by smoothing the double descent peaks).

This talk will present theoretical results on subagging of regularized M-estimators under proportional asymptotics, where the sample size, feature size, and subsample sizes all grow with fixed limiting ratios. Our analysis incorporates risk asymptotics, optimal (oracle) tuning properties, and data-dependent tuning. Of independent interest, in the non-ensemble setting (M=1), our analysis also establishes convergence of trace‐based degrees-of-freedom functionals, extending previous results for square loss and ridge, lasso regularizers.

This is joint work with the following collaborators (in alphabetical order): Pierre Bellec, Jin-Hong Du, Takuya Koriyama, and Kai Tan.