Latent Variable Models

Latent variable models assume that observed data is generated from hidden (unobserved) variables, and learning means inferring those hidden variables and their relationship to the data.

$$P_{\theta }(x)=\sum _z{P}_{\theta }(x,z)\text{or}{P}_{\theta }(x)={\int }_z{P}_{\theta }(x,z)dz$$

Here $z$ is an latent/unobserved/hidden random variable. Typically $z$ is jointly estimated along with the model parameters $\theta$. For each $x_i\in D$ we assume that there exists a corresponding $z_i$.

1. If $z$ is discrete, $z\in \{z_1,z_2,\dots ,z_m\}$. Because for every $x_i\in D$ there would be a corresponding $z_i$, we can create $m$ such buckets for each $x_i$. This is basically what K-Means Clustering and Gaussian Mixture Models aim to do.
2. If $z$ is continuous, $x\in {\mathbb{R}}^d,z\in {\mathbb{R}}^k$. Typically $k<<d$. Here $z_i|x_i$ represents a feature vector corresponding to the given $x_i$.

Principle for Learning LVMs

Suppose we have a dataset $D=\{x_i{\}}_{i=1}^n$ and a Latent Variable Model $P_{\theta }(x)={\int }_{Z}P(x,z)dz$. Our goal here would be to estimate the model parameter $\theta$ given $D$. This can be done by minimizing the KL Divergence.

$$\begin{array}{rl}{\theta }^{\ast }& =\arg \underset{\theta }{\min }{D}_{KL}(P_x||P_{\theta })\\ & =\arg \underset{\theta }{\min }[{\int }_X{P}_X(x)log\left(\frac{P_X(x)}{P_{\theta }(x)}\right)dx]\\ & =\arg \underset{\theta }{\min }[{\int }_X{P}_X(x)log\left(P_X(x)\right)dx-{\int }_X{P}_X(x)log\left(P_{\theta }(x)\right)dx]\\ & =\arg \underset{\theta }{\min }[{\cancel{{\int }_X{P}_X(x)log\left(P_X(x)\right)dx}}^{\text{Independent of }\theta }-{\int }_X{P}_X(x)log\left(P_{\theta }(x)\right)dx]\\ & =\arg \underset{\theta }{\min }[-{\int }_X{P}_X(x)log\left(P_{\theta }(x)\right)dx]\\ & =\arg \underset{\theta }{\max }[{\int }_X{P}_X(x)log\left(P_{\theta }(x)\right)dx]\\ & =\arg \underset{\theta }{\max }\underset{P_X}{\mathbb{E}}\left[log\left(P_{\theta }(x)\right)\right]\end{array}$$

Here $log\left(P_{\theta }(x)\right)$ is the log-likelihood function of $P_{\theta }(x)$. Thus this optimization problem is called Maximum Likelihood Estimation. Instead of solving the entire optimization problem at once, let’s just consider the following,

$$\begin{array}{rl}l(\theta )& =log{P}_{\theta }(x)\\ & =log{\int }_Z{P}_{\theta }(x,z)dz\\ & =log{\int }_Z{P}_{\theta }(x,z)\frac{q(z|x)}{q(z|x)}dz\\ & =log{\int }_{Z}q(z|x)\frac{P_{\theta }(x,z)}{q(z|x)}dz\\ & =log\underset{q(z|x)}{\mathbb{E}}\left[\frac{P_{\theta }(x,z)}{q(z|x)}\right]\end{array}$$

By Jensen’s Inequality we know that $log\mathbb{E}[f(x)]\ge \mathbb{E}log[f(x)]$. So applying this on the above equation for $l(\theta )$ we get,

$$\begin{array}{rl}l(\theta )& =log\underset{q(z|x)}{\mathbb{E}}\left[\frac{P_{\theta }(x,z)}{q(z|x)}\right]\\ & \ge \underset{q(z|x)}{\mathbb{E}}log\left[\frac{P_{\theta }(x,z)}{q(z|x)}\right](\text{denoted as }{J}_{\theta }(q))\end{array}$$

Here $J_{\theta }(q)$ is called the Evidence Lower Bound (ELBO). $J_{\theta }(q)$ is function of both the model parameters $\theta$ and the density on $z$, $q(z|x)$. $q(z|x)$ is called the Variational Latent Posterior. Similar to VDM, here too we maximize a lower-bound of a value in order to optimize it.

$$\begin{array}{rl}{\theta }^{\ast },q^{\ast }& =\arg \underset{\theta ,q}{\max }{J}_{\theta }(q)\\ \text{where, }{J}_{\theta }(q)& =\underset{q(z|x)}{\mathbb{E}}\left[log\left[\frac{P_{\theta }(x,z)}{q(z|x)}\right]\right]\end{array}$$

Gaussian Mixture Models (GMM)

In GMMs $z$ is discrete, $z\in \{1,2,\dots ,M\}$.

$$\begin{array}{rl}{P}_{\theta }(x)& =\sum _Z{P}_{\theta }(x,z)\\ & =\sum _{j=1}^M{P}_{\theta }(z=j)P_{\theta }(x|z=j)\end{array}$$

In a GMM, $P_{\theta }(z=j)={\alpha }_j$, $P_{\theta }(x|z=j)=\mathcal{N}(x;{\mu }_j,{\Sigma }_j)$.

$$\begin{array}{rl}{P}_{\theta }(x)& =\sum _{j=1}^M{P}_{\theta }(z=j)P_{\theta }(x|z=j)\\ & =\sum _{j=1}^M{\alpha }_j\cdot \mathcal{N}(x;{\mu }_j,{\Sigma }_j)\end{array}$$

Parameters of a GMM are,

$$\theta =\{{\alpha }_1,{\alpha }_2,\dots ,{\mu }_1,{\mu }_2,\dots ,{\Sigma }_1,{\Sigma }_2,\dots \}$$

Here $x\in {\mathbb{R}}^d$, ${\mu }_j\in {\mathbb{R}}^d$, and ${\Sigma }_j\in {\mathbb{R}}^{d\times d}$ and $0\le {\alpha }_j\le 1,{\sum }_{i=1}^m{\alpha }_j=1$. Since our goal is to estimate $\theta ,q$ via ELBO optimization, we can use the Expectation Maximization Algorithm (EM Algorithm) which updates both $\theta ,q$ alternatively.

$$\begin{array}{rlr}\text{For }t=1\text{ to }T:& & \\ q_{t+1}^{\ast }& =\arg \underset{q}{\max }{J}_{{\theta }^t}(q)& (\text{with }{\theta }_t\text{ as constant})\\ {\theta }_{t+1}^{\ast }& =\arg \underset{\theta }{\max }{J}_{\theta }(q_{t+1})& (\text{with }{q}_{t+1}\text{ as constant})\end{array}$$

It can be shown that EM ensures that $l({\theta }_{t+1})\ge l({\theta }_t)$. This doesn’t ensure that the likelihood function will keep on increasing, but it ensures that it won’t decrease as the parameters get updated.

Applying EM algorithm for the GMM, it can be shown analytically (try once) that,

$$\begin{array}{rlr}{q}_{t+1}^{\ast }& =\arg \underset{q}{\max }{J}_{\theta }(q)& \\ & =P_{\theta }(z|x)& \text{(Excercise to show)}\\ & =\frac{P_{\theta }(x|z)\cdot P_{\theta }(z)}{P_{\theta }(x)}& \\ & =\frac{\mathcal{N}(x;{\mu }_j,{\Sigma }_j)\cdot {\alpha }_j}{{\sum }_{j=1}^{\mu }\mathcal{N}(x;{\mu }_j,{\Sigma }_j)\cdot {\alpha }_j}& \left(P_{\theta }(x)=\sum _{j=1}^M{P}_{\theta }(z_j)P_{\theta }(x|z_j)\right)\\ {\theta }_{t+1}^{\ast }& =\arg \underset{\theta }{\max }{J}_{\theta }(q)& \end{array}$$

ELBO can be optimized for an LVM using the EM algorithm, if $P_{\theta }(z|x)$ can be computed.

Variational Auto-Encoders (VAE)