Differential Equations in Deep Generative Modeling
Steven Strogatz: “Since Newton, mankind has come to realize that the laws of physics are always expressed in the language of differential equations.”
Mathematically, a differential equation is an equation that relates one or more unknown functions and their derivatives, like
\[\frac{\mathrm{d}y}{\mathrm{d}x}=f(x,y),\]where $y$ is an unknown function of $x$ and $f$ is a given function. Such equations describe the rules by which the state of a system changes—changes (motion, growth, decay, or diffusion).
Meanwhile, generative models, such as autoregressive models and variational autoencoders (VAEs), can generally be characterized by the following paradigm: sample $X_0$ from a simple initial distribution $p_\text{init}$ (a Gaussian distribution $\mathcal{N}(0, I_d)$ for example) and transform it into $X_1$, which follows the dataset distribution $p_\text{data}$.
In this blog, we discuss how such transformations can be modeled using specific types of differential equations, namely ordinary differential equations (ODEs) and stochastic differential equations (SDEs), corresponding to flow models and diffusion models, respectively. Subsequently, we discuss how deep neural networks are employed to determine the parameters of these models, which is exactly the process of training them.
Comments