Operator SVD with Neural Networks via Nested Low-Rank Approximation

Abstract: Top-$L$ eigenvalue decomposition (EVD) of a given linear operator, or finding its top-$L$ eigenvalues and eigenfunctions, is a fundamental task in many machine learning and scientific simulation problems.For high-dimensional eigenvalue problems, training neural networks to parameterize the eigenfunctions is considered as a promising alternative to the classical numerical linear algebra techniques.
While several optimization frameworks have been proposed in this parametric approach, all the existing proposals either use an ad-hoc regularization to obtain orthogonal eigenfunctions and/or inherently suffer with biased gradient estimates.
In this talk, I will present a new optimization framework based on the low-rank approximation characterization of a truncated singular value decomposition (SVD), accompanied with a technique called nesting for correctly learning the top-$L$ singular- value and functions up to degeneracy. Top-$L$ EVD can be performed as a special case. The proposed optimization framework is easy to implement with off-the-shelf gradient-based optimization algorithms, since (1) it is based on an unconstrained optimization problem that naturally admits an unbiased gradient estimator, and (2) it works without any extra orthonormalization steps and regularization terms. The proposed optimization framework can be used in a variety of application scenarios, and I will briefly discuss its application in machine learning and computational physics.

Speaker bio: Jongha (Jon) Ryu is a postdoctoral associate at Research Laboratory of Electronics (RLE) hosted by Prof. Gregory W. Wornell. His research in general aims to develop efficient, reliable, and robust machine learning algorithms with provable performance guarantees, especially with inspirations from information theory. He is currently interested in representation learning, generative models, and learning with uncertainty.