If you ask deep learning people "what is the best image generation model now", many of them would probably say "generative adversarial networks" (GAN). The original paper describes an adversarial process that asks the generator and the discriminator to fight against each other, and many people including myself like this intuition. What lessons can we learn from GAN for better approximate inference (which is my thesis topic)? I need to rewrite the framework in the language I'm comfortable with, hence I decided to put it on this blog post as a research note.

Suppose there's a machine that can show you some images. This machine flips a coin to determine what it will show to you. For heads, the machine shows you a gorgeous paint from human artists. For tails, it shows you a forged famous paint. Your job is to figure out whether the shown image is "real" or not.

Let me be precise about it using math language. Assume $s \in \{0, 1\}$ denotes the outcome of that coin flip, where 0 represents tails and 1 represents heads. The coin could possibly be a bent one so let's say your prior belief of the outcome is $\tilde{p}(s = 0) = \pi$. Then the above process is summarized as the following:

$$s \sim \tilde{p}(s), x \sim \tilde{p}(x|s),$$

$$\tilde{p}(x|s = 0) = p(x), \tilde{p}(x|s = 1) = p_D(x).$$

Here I use $p_D(x)$ to describe the unknown data distribution and $p(x)$ as the actual generative model we want to learn. As discussed, the generative model don't want you to know about $\pi$, in math this is achieved by minimizing the mutual information

$$\mathrm{I}[s; x] = \mathrm{KL}[\tilde{p}(s, x) || \tilde{p}(s) \tilde{p}(x)].$$

This is quite intuitive: $\mathrm{I}[s; x] = 0$ iff. $p_D(x) = p(x)$, and in this case observing an image $x$ tells you nothing about the outcome of the coin flip $s$. However computing this mutual information requires evaluating $p(x)$ which is generally intractable. Fortunately we can use variational approximation similar to the variational information maximization algorithm. In detail we construct a lower-bound by subtracting the KL divergence:

$$\mathcal{L}[p; q] = \mathrm{I}[s; x] - \mathbb{E}_{\tilde{p}(x)}[\mathrm{KL}[\tilde{p}(s|x) ||q(s|x)] \\ = \mathrm{H}[s] + \mathbb{E}_{\tilde{p}(s, x)}[\log q(s|x)].$$

Let's have a closer look at the equations. From the KL term we can interpret $q(s|x)$ as the approximated posterior of $s$ under the augmented model $\tilde{p}$. But more interestingly $q$ can also be viewed as the discriminator in the GAN framework. To see this, we expand the second term as

$$\tilde{p}(s = 0) \mathbb{E}_{p(x)}[\log (1 - q(s = 1|x))] + \tilde{p}(s = 1) \mathbb{E}_{p_D(x)}[\log q(s = 1|x)].$$

Notice something familiar with? Indeed when we pick the prior as $\pi = 0.5$, the lower-bound $\mathcal{L}[p; q]$ is exactly GAN's objective function (up to scaling and adding constants), and the mutual information $\mathrm{I}[s; x]$ becomes the Jensen-Shannon divergence $\mathrm{JS}[p(x)||p_D(x)]$, which GAN is actually minimizing.

To summarize, GAN can be viewed as an augmented generative model which is trained by minimizing mutual information. This augmentation is smart in the sense that it uses label-like information $s$ that can be obtained for free, which introduces supervision signal to help unsupervised learning.

How useful is this interpretation? Well, in principle we can carefully design an augmented model and learn it by minimizing L2 loss/KL divergence/mutual information... you name it. But more interestingly, we can then do (automatic) divergence/algorithm selection as model selection in the "augmented" space. In the GAN example setting $\pi$ determines which JS divergence variant we use in training, e.g. see this paper. I'm not sure how useful it would be, but we can also learn $\pi$ by say maximum likelihood, or even treat $\pi$ as a latent variable and put a Beta prior on it.

Recently I started to think about automatic algorithm selection. Probably because I'm tired of my reviewers complaining on my alpha-divergence papers "I don't know how to choose alpha and you should give us a guidance". Tom Minka gives an empirical guidance in his divergence measure tech report, and same for us in recent papers. I know this is an important but very difficult topic for research, but at least not only myself have thought about it, e.g. in this paper the authors connected beta-divergences to tweedie distributions and performed approximate maximum likelihood to select beta. Another interesting paper in this line is the "variational tempering" paper which models the annealing temperature in a probabilistic model as well. I like these papers as the core idea is very simple: we should also use probabilistic modeling for algorithmic parameters. Perhaps this also connects to Bayesian optimization but I'm gonna stop here as the note is already a bit too long.

EDIT 06/08/16:

1. I briefly presented this MI interpretation (also extended to ALI) to the MLG summer reading group and you can see some notes here.
2. Ferenc also posted a discussion of infoGAN with mutual information interpretation which is very nice. He also has a series of great blog posts on GANs that I wish I could have read them earlier!

Here's a general case that the 1-step learning will failed to achieve the optimal parameters, inspired by [1].

Consider the exponential family model

$$P(x; \theta) = \frac{1}{Z(\theta)} exp(\theta^Tf(x)),$$

$$\exists k' \in [1, N] s.t. f_{k'}(x) = f_{k'}(T(x)), \forall x \in \{x_i\}.$$

The statement indicates that the performance of the one-step learning depends on the model distribution and the transition in each step of the Markov chain. However, though the failure above will be addressed if provided enough training data points, in some cases the parameters will never converge or converge to the other points rather than the ML result. See the cases of noisy swirl operator and noisy star trek operator in [1] for details. But fortunately, training energy models with sampling from factored likelihood and posterior in each step yields good results with CD-k, which will be explained below.

Let's consider the justification of the CD-k algorithm Rich and I discussed in our meeting, to answer the questions I raised in the last research note. We learn the energy models by the maximum likelihood method, i.e. maximize

$$\langle log P(x)\rangle_{P_0}$$

$$max \langle log P(x)\rangle_{P_0} \Leftrightarrow min -H(P_0) - \langle log P(x)\rangle_{P_0},$$

$$P(x) = \frac{1}{Z(\theta)} exp(-E(x; \theta))$$

$$- \frac{\partial}{\partial \theta} \langle log P(x)\rangle_{P_0} = \langle \frac{\partial}{\partial \theta} E(x; \theta)\rangle_{P_0} - \langle \frac{\partial}{\partial \theta} E(x; \theta)\rangle_P.$$

In the RBM case, we have the model joint distribution

$$P(x, h|\theta) = \frac{1}{Z(\theta)} exp(-E(x, h; \theta)),$$

$$- \frac{\partial}{\partial \theta} \langle log P(x)\rangle_{P_0} = \langle \frac{\partial}{\partial \theta} E(x, h; \theta)\rangle_{P_0(x)P(h|x)} - \langle \frac{\partial}{\partial \theta} E(x; \theta)\rangle_{P(x, h|\theta)}.$$

However, the expectation is intractable, and in practice we sample the training set $\{ x_i\}_{i = 1}^{N}$ from $P_0$. In this case we approximate the first term of the derivative by $\frac{1}{N} \sum_{i = 1}^{N} \frac{\partial}{\partial \theta} E(x_i, h; \theta)$, where $h$ equals to, or is sampled from, the conditional distribution $P(h|x_i; \theta)$. For better generalization, we don't want the ML algorithm to minimize the KL-divergence exactly to 0, as P will be the average of Dirac functions

$$P(x) = \frac{1}{N}\sum_{i=1}^{N}\delta(x - x_i)$$

which makes the probability be zero at the points other than those in the training set. That's why the approximation of the ML optimum can help to smooth the model distribution.

Also notice that the derivative of the log-likelihood of the RBM contains the expectation of the energy function over both $x$ and $h$ with the joint distribution $P(x, h|\theta)$, which is also intractable. In general we use the samples from some MCMC method, e.g. Gibbs sampling, to approximate that derivative. Without loss of generality we assume $\{x, h\}$ have discrete states and the Markov chain is aperiodic, then consider the sampling conditional probability $P(h|x)$ and $P(x|h)$, the Gibbs chain will converge to the equilibrium distribution $P_\infty$ and $P_\infty = P$. To specify the statement, consider

$$P_1(x_2, h_2) = P(h_2|x_2)P(x_2|h_1)P(h_1|x_1)$$

References

[1] David J. C. MacKay, Failures of the One-Step Learning Algorithm, 2001.