Latent Variable Models

Latent variable models assume that observed data is generated from hidden (unobserved) variables, and learning means inferring those hidden variables and their relationship to the data.

$$P_{\theta }(x)=\sum _z{P}_{\theta }(x,z)\text{or}{P}_{\theta }(x)={\int }_z{P}_{\theta }(x,z)dz$$

Here $z$ is an latent/unobserved/hidden random variable. Typically $z$ is jointly estimated along with the model parameters $\theta$. For each $x_i\in D$ we assume that there exists a corresponding $z_i$.

1. If $z$ is discrete, $z\in \{z_1,z_2,\dots ,z_m\}$. Because for every $x_i\in D$ there would be a corresponding $z_i$, we can create $m$ such buckets for each $x_i$. This is basically what K-Means Clustering and Gaussian Mixture Models aim to do.
2. If $z$ is continuous, $x\in {\mathbb{R}}^d,z\in {\mathbb{R}}^k$. Typically $k<<d$. Here $z_i|x_i$ represents a feature vector corresponding to the given $x_i$. In doing so, the model is attempting to find a “hidden space”, usually lower dimension, in which the data lives.

TL;DR - If the latent variable is discrete, then the model can be used for clustering. If the latent variable is continuous, then the model can be used for this feature extraction task.

Principle for Learning LVMs

Suppose we have a dataset $D=\{x_i{\}}_{i=1}^n$ and a Latent Variable Model $P_{\theta }(x)={\int }_{Z}P(x,z)dz$. Our goal here would be to estimate the model parameter $\theta$ given $D$. This can be done by minimizing the KL Divergence.

$$\begin{array}{rl}{\theta }^{\ast }& =\arg \underset{\theta }{\min }{D}_{KL}(P_x||P_{\theta })\\ & =\arg \underset{\theta }{\min }[{\int }_X{P}_X(x)log\left(\frac{P_X(x)}{P_{\theta }(x)}\right)dx]\\ & =\arg \underset{\theta }{\min }[{\int }_X{P}_X(x)log\left(P_X(x)\right)dx-{\int }_X{P}_X(x)log\left(P_{\theta }(x)\right)dx]\\ & =\arg \underset{\theta }{\min }[{\cancel{{\int }_X{P}_X(x)log\left(P_X(x)\right)dx}}^{\text{Independent of }\theta }-{\int }_X{P}_X(x)log\left(P_{\theta }(x)\right)dx]\\ & =\arg \underset{\theta }{\min }[-{\int }_X{P}_X(x)log\left(P_{\theta }(x)\right)dx]\\ & =\arg \underset{\theta }{\max }[{\int }_X{P}_X(x)log\left(P_{\theta }(x)\right)dx]\\ & =\arg \underset{\theta }{\max }\underset{P_X}{\mathbb{E}}\left[log\left(P_{\theta }(x)\right)\right]\end{array}$$

Here $log\left(P_{\theta }(x)\right)$ is the log-likelihood function of $P_{\theta }(x)$. Thus this optimization problem is called Maximum Likelihood Estimation. Because expectation is a linear function and we are calculating the expectation w.r.t $P_X$ not $P_{\theta }$, we can try to maximize each $log$ term independently and then take the expectation of all to get the final output. So let’s just consider the following,

$$\begin{array}{rl}l(\theta )& =log{P}_{\theta }(x)\\ & =log{\int }_Z{P}_{\theta }(x,z)dz\\ & =log{\int }_Z{P}_{\theta }(x,z)\frac{q(z|x)}{q(z|x)}dz\\ & =log{\int }_{Z}q(z|x)\frac{P_{\theta }(x,z)}{q(z|x)}dz\\ & =log\underset{q(z|x)}{\mathbb{E}}\left[\frac{P_{\theta }(x,z)}{q(z|x)}\right]\end{array}$$

By Jensen’s Inequality we know that $log\mathbb{E}[f(x)]\ge \mathbb{E}log[f(x)]$. So applying this on the above equation for $l(\theta )$ we get,

$$\begin{array}{rl}l(\theta )& =log\underset{q(z|x)}{\mathbb{E}}\left[\frac{P_{\theta }(x,z)}{q(z|x)}\right]\\ & \ge \underset{q(z|x)}{\mathbb{E}}log\left[\frac{P_{\theta }(x,z)}{q(z|x)}\right](\text{denoted as }{J}_{\theta }(q))\end{array}$$

Here $l(\theta )$ is called the evidence and thus $J_{\theta }(q)$ is called the Evidence Lower Bound (ELBO). $J_{\theta }(q)$ is function of both the model parameters $\theta$ and the density on $z$, $q(z|x)$. $q(z|x)$ is called the Variational Latent Posterior. Similar to VDM, here too we maximize a lower-bound of a value in order to optimize it.

$$\begin{array}{rl}{\theta }^{\ast },q^{\ast }& =\arg \underset{\theta ,q}{\max }{J}_{\theta }(q)\\ \text{where, }{J}_{\theta }(q)& =\underset{q(z|x)}{\mathbb{E}}\left[log\left[\frac{P_{\theta }(x,z)}{q(z|x)}\right]\right]\end{array}$$

Gaussian Mixture Models (GMM)

In GMMs $z$ is discrete, $z\in \{1,2,\dots ,M\}$.

$$\begin{array}{rl}{P}_{\theta }(x)& =\sum _Z{P}_{\theta }(x,z)\\ & =\sum _{j=1}^M{P}_{\theta }(z=j)P_{\theta }(x|z=j)\end{array}$$

In a GMM, $P_{\theta }(z=j)={\alpha }_j$, $P_{\theta }(x|z=j)=\mathcal{N}(x;{\mu }_j,{\Sigma }_j)$.

$$\begin{array}{rl}{P}_{\theta }(x)& =\sum _{j=1}^M{P}_{\theta }(z=j)P_{\theta }(x|z=j)\\ & =\sum _{j=1}^M{\alpha }_j\cdot \mathcal{N}(x;{\mu }_j,{\Sigma }_j)\end{array}$$

Parameters of a GMM are,

$$\theta =\{{\alpha }_1,{\alpha }_2,\dots ,{\mu }_1,{\mu }_2,\dots ,{\Sigma }_1,{\Sigma }_2,\dots \}$$

Here $x\in {\mathbb{R}}^d$, ${\mu }_j\in {\mathbb{R}}^d$, and ${\Sigma }_j\in {\mathbb{R}}^{d\times d}$ and $0\le {\alpha }_j\le 1,{\sum }_{i=1}^m{\alpha }_j=1$ (convex combination of Gaussian Distributions). Since our goal is to estimate $\theta ,q$ via ELBO optimization, we can use the Expectation Maximization Algorithm (EM Algorithm) which updates both $\theta ,q$ alternatively.

$$\begin{array}{rlr}\text{For }t=1\text{ to }T:& & \\ q_{t+1}^{\ast }& =\arg \underset{q}{\max }{J}_{{\theta }^t}(q)& (\text{with }{\theta }_t\text{ as constant})\\ {\theta }_{t+1}^{\ast }& =\arg \underset{\theta }{\max }{J}_{\theta }(q_{t+1})& (\text{with }{q}_{t+1}\text{ as constant})\end{array}$$

It can be shown that EM ensures that $l({\theta }_{t+1})\ge l({\theta }_t)$. This doesn’t ensure that the likelihood function will keep on increasing, but it ensures that it won’t decrease as the parameters get updated.

Applying EM algorithm for the GMM, it can be shown analytically (try once) that,

$$\begin{array}{rlr}{q}_{t+1}^{\ast }& =\arg \underset{q}{\max }{J}_{\theta }(q)& \\ & =P_{{\theta }_t}(z|x)& \text{(Excercise to show)}\\ & =\frac{P_{{\theta }_t}(x|z)\cdot P_{{\theta }_t}(z)}{P_{{\theta }_t}(x)}& \\ & =\frac{\mathcal{N}(x;{\mu }_j,{\Sigma }_j)\cdot {\alpha }_j}{{\sum }_{j=1}^{\mu }\mathcal{N}(x;{\mu }_j,{\Sigma }_j)\cdot {\alpha }_j}& \left(P_{\theta }(x)=\sum _{j=1}^M{P}_{\theta }(z_j)P_{\theta }(x|z_j)\right)\\ {\theta }_{t+1}^{\ast }& =\arg \underset{\theta }{\max }{J}_{\theta }(q)& \end{array}$$

ELBO can be optimized for an LVM using the EM algorithm, provided $P_{\theta }(z|x)$ can be computed. If $P_{\theta }(z|x)$ can’t be computed, then EM fails.

Variational Auto-Encoders (VAE)