236610 – Diffusion Diffusion Diffusion – Winter 2022/2023

Diffusion Diffusion Diffusion (236610)

Winter 2022/2023

Given as an “Advanced Topics in CS” Course

Last date of update: September 7th 2022

Note: Registration to this course is closed, as we have reached the limit of our capacity.

Team	Lecturer: Prof. Michael Elad Teaching Assistant: Bahjat Kawar
Credit	2 points
Format	This course will be given in online format over Zoom Participation (including free listeners) will be restricted to Technion’s students after enlisting with the lecturer Attendance is not mandatory and will not be checked. However, an exam will verify your familiarity with the content covered in class
Pre-requisites	236200 or 236201 (Introduction to Data Processing and Representation) or 046200 (Processing and Analysis of Images) A provable background in programming in deep-learning – wither an advanced course on this topic or a project
Parallel Courses	048954 (Statistical Methods in Image Processing) is NOT considered as parellel, but students should avoid double-deeping their projects
Course Description	A fascinating machine learning topic in recent years touches on the grand challenge of synthesizing content out of thin air. This includes the creation of images, paragraphs, short videos, music pieces, and more. In most cases, this synthesis goes hand in hand with the notion of GAN’s – Generative Adversarial Networks, although other techniques for such synthesis do exist, such as VAE, energy-based methods, Normalizing flow, and more. The entry of diffusion techniques as a competitive alternative to all the above brought a revolution. This fresh line of work that already grows exponentially offers iterative algorithms that start with a Gaussian iid noise and end with the desired synthesized content. A key to the success of these techniques is an access to the score function of the data distribution, which can be obtained by simple denoisers. This makes diffusion techniques quite appealing and effective. In this course we will study this field by surverying the existing literature, exploring its extensions and prospects, and by diving into this field’s mathematical foundations.
Course Structure	After few (2-3) lectures by the lecturer, we will concentrate on students’ lectures on their assigned projects, covering various recent papers in this domain. Each student participating in the course will be assigned with a paper (or more, depending on the content) as the grounds for their project. The project itself will include the following activities: Phase 1: Reading the paper(s) and understanding their content; Preparing a slide show to present the above – the slides should be cleared by the lecturer before being presented; Phase 2: Implementing the algorithms suggested in the paper(s), if any; Exploring possible extensions to the work suggested; Updating the slides to include the additional work; Presenting the modified slides to the lecturer and/or the TA; and Optional (bonus grade): Issuing a report in a paper format to summarize all the above.
Grading Policy	40% of the grade will be dictated by the quality of the presentation created and the lecture given. 40% of the grade will be dictated by the quality of the work that goes beyond the paper’s content. 20% of the grade will be dictated by an exam (20 questions) on general familiarity with the content presented in class by the lecturer and the students. This exam is likely to take place on the first week of the Spring Semester, after all students presented their lecture. 15% bonus will be given to those who submit a paper-format report of their work – relevant only after getting an approval from the lecturer based on the extra content created.
Choosing a Project’s Paper	Please choose your paper for the project from THIS LIST. Note: You have to notify the teaching team (both Bahjat and Miki) of the paper you have chosen. If it has been already taken, you will have to choose again. You are welcome to suggest your own choice of paper outside this list, but it should be (i) relevant to the course; (ii) with open source code, or a reasonable alternative to it; and (iii) not something too basic that we have intend to present in class.
Tentative Syllabus	Background: The role of priors in image processing The classical era of priors The migration to deep-learning based alternatives – GANs, VAE, Flow methods, Energy-based techniques, and more. Diffusion methods: Mathematical foundations (Stochastic Differential Equations, Score Approximation, Langevin and other samplers, relation to RED) Diffusion Methods – Variations of algorithms (one-sided flow, two sided flow, accelerations) Diffusion Methods – Building blocks (MMSE Image Denoisers, GAN-based Score approximation, alternatives to the score) Diffusion Methods – Applications (Image synthesis, Audio synthesis, Solving inverse problems, Conditional sampling, IMAGEN and related methods, Image Compression, …)

Preliminary Resources:

Review of latest Score Based Generative Modeling papers
Yang Song’s blog on Generative Models and Score-Based Diffusion
What Are Diffusion Models? A blog by Lili Weng
CVPR tutorial on diffusion-based generative modeling: Slides and Video
What are diffusion models? A video by Ari Seff (Princeton)

People to Follow:

Stefano Ermon (Stanford CS)
Yang Song (OpenAI)
Arash Vahdat (NVIDIA)
Diederick P. Kingma (Google)
Prafulla Dhariwal (OpenAI)
Tim Salimans (Google)