Function Space Diffusion for Video Modeling

Add to your list(s) Download to your calendar using vCal

• Nikola Kovachki (None / Other)
• Thursday 18 July 2024, 14:30-15:30
• External.

If you have a question about this talk, please contact nobody.

DMLW01 - International workshop on diffusions in machine learning: foundations, generative models, and optimisation

We present a generalization of score-based diffusion models to function space by perturbing functional data via a Gaussian process at multiple scales. We obtain an appropriate notion of score by defining densities with respect to Guassian measures and generalize denoising score matching. We then define the generative process by integrating a function-valued Langevin dynamic. We show that the corresponding discretized algorithm generates samples at a fixed cost that is independent of the data discretization. As an application for such a model, we formulate video generation as a sequence of joint inpainting and interpolation problems defined by frame deformations. We train an image diffusion model using Gaussian process inputs and use it to solve the video generation problem by enforcing equivariance with respect to frame deformations. Our results are state-of-the-art for video generation using models trained only on image data.

This talk is part of the Isaac Newton Institute Seminar Series series.

This talk is included in these lists:

• All CMS events
• External
• Featured lists
• INI info aggregator
• Isaac Newton Institute Seminar Series
• School of Physical Sciences
• bld31

Note that ex-directory lists are not shown.