The Role of Information Measures on the Regularization of Empirical Risk Minimization

Add to your list(s) Download to your calendar using vCal

• Dr Iñaki Esnaola, University of Sheffield
• Wednesday 29 November 2023, 14:00-15:00
• MR5, CMS Pavilion A.

If you have a question about this talk, please contact Prof. Ramji Venkataramanan.

The empirical risk minimization problem with relative entropy regularization (ERM-RER) is presented considering that the reference measure is a $\sigma$-finite measure instead of a probability measure. This generalization allows for a larger degree of flexibility in the incorporation of prior knowledge over the set of models. We discuss the interplay of the regularization parameter, the reference measure, the risk measure, and the expected empirical risk induced by the solution of the ERM -RER problem, which is proved to be unique. We show that the expectation of the sensitivity is upper bounded, up to a constant factor, by the square root of the lautum information between the models and the datasets. Using these tools, dataset aggregation is studied and different figures of merit to evaluate the generalization capabilities of ERM -RER are introduced. For arbitrary datasets and parameters of the ERM -RER solution, a connection between Jeffrey’s divergence, training, and test error is established. We conclude by extending the results to $f$-divergence regularization by obtaining a closed form expression for the solution under mild assumptions on the structure of the regularizer. This analytical solution is leveraged to characterize the sensitivity of the resulting supervised learning problem and we evaluate the solution for specific regularizers arising in estimation, high-dimensional statistics, and hypothesis testing.

This talk is part of the Information Theory Seminar series.

This talk is included in these lists:

• All CMS events
• All Talks (aka the CURE list)
• CMS Events
• DPMMS Lists
• DPMMS info aggregator
• DPMMS lists
• Hanchen DaDaDash
• Information Theory Seminar
• Interested Talks
• MR5, CMS Pavilion A
• School of Physical Sciences
• Statistical Laboratory info aggregator
• bld31

Note that ex-directory lists are not shown.