Modèles génératifs pour l’image
B. GALERNE, V. DE BORTOLI
Image processingModelling

Prè-requis

Les bases d’un M1 de mathématiques appliquées en probabilités, statistiques et optimisation.

Objectif du cours

The goal of this course is to give an overview on existing methods for generative modeling with applications to images. The first part of the course will focus on modern image models such as  Variational AutoEncoders (VAEs), Generative Adversarial Networks (GAN) and Normalizing Flows (NFs). Applications of these models for conditional image generation will be discussed. The rest of the course will be dedicated to the study of a new contender in generative modeling called diffusion models or Score-Based Generative Models (SGM). We will study the mathematical aspects (Time reversal of stochastic processes, links with Regularized Optimal Transport and control) and practical aspects of these algorithms. In particular we will introduce the core concepts of practical implementations like DDIM (Denoising Implicit Diffusion Models), LDM (Latent Diffusion Models) and their application to inverse problem and text-to-image synthesis through the lens of StableDiffusion.

Classes will be given in French but slides will be written in English.

Organisation des séances

9 séances de 3h

Mode de validation

1 Devoir maison obligatoire à mi-cours
1 Projet avec étude d’un article à la fin du cours

Références

Generative Adversarial Networks (Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio, 2014)

Auto-Encoding Variational Bayes (Kingma and Welling, 2014)

An Introduction to Variational Autoencoders (Kingma and Welling, 2019)

Wasserstein GAN (Martin Arjovsky, Soumith Chintala, Léon Bottou, 2017)

A style-based generator architecture for generative adversarial networks (Karras, T., Laine, S., and Aila, T., 2019)

Image-To-Image Translation With Conditional Adversarial Networks (Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A. Efros, 2019)

Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network (Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, Wenzhe Shi, 2017)

Generative Modeling by Estimating Gradients of the Data Distribution (Yang Song, Stefano Ermon, 2019)

Score-Based Generative Modeling through Stochastic Differential Equations (Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, Ben Poole, 2021)

Denoising Diffusion Probabilistic Models (Jonathan Ho, Ajay Jain, Pieter Abbeel, 2020)

Diffusion Schrödinger Bridge with Applications to Score-Based Generative Modeling (Valentin De Bortoli, James Thornton, Jeremy Heng, Arnaud Doucet, 2021)

Denoising Diffusion Implicit Models (Jiaming Song, Chenlin Meng, Stefano Ermon, 2020)

Score-based Generative Modeling in Latent Space (Arash Vahdat, Karsten Kreis, Jan Kautz, 2021)

Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, Mohammad Norouzi, 2022)

High-Resolution Image Synthesis with Latent Diffusion Models (Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, Björn Ommer, 2022)

 

Thèmes abordés

Variational Auto-Encoders

Generative Adversarial Networks

Conditional image generation

Score-based Generative Modelling

Regularized Optimal Transport

Stochastic Processes and their discretizations

 

Les intervenants

Bruno Galerne

(Institut Denis Poisson, Université d'Orléans)

Valentin De Bortoli

(CNRS et DI ENS)

voir les autres cours du 2nd semestre