An Exposition on the Propriety of Restricted Boltzmann Machines

What is this?

Used for image classification. Each image pixel is a node in the visible layer. The output creates features, passed to supervised learning.

Joint Distribution

Let \(x = \{h_1, ..., h_H, v_1, ...,v_V\}\) represent the states of the visible and hidden nodes in an RBM. Then the probability each node taking the value corresponding to \(x\) is:

\[ f_{\theta} (x) = \frac{\exp\left(Q(x)\right)}{\sum\limits_{x \in \mathcal{X}}\exp\left(Q(x)\right)} \]

Where \(Q(x) = \sum\limits_{i = 1}^V \sum\limits_{j=1}^H \theta_{ij} v_i h_j + \sum\limits_{i = 1}^V\theta_{v_i} v_i + \sum\limits_{j = 1}^H\theta_{h_j} h_j\) denotes the neg-potentional function of the model, having support set \(\mathcal{S}\).

Deep learning

Claimed ability to learn “internal representations that become increasingly complex” (Salakhutdinov and Hinton 2009), used in classification problems.

Why do I care?

Current heuristic fitting methods seem to work for classification. Beyond classification, RBMs are generative models:

To generate data from an RBM, we can start with a random state in one of the layers and then perform alternating Gibbs sampling. (Hinton, Osindero, and Teh 2006)

Can we fit a model that generates data that looks like data?

Near-degeneracy

The highly flexible nature of the RBM (\(H + V + HV\) parameters) makes three characteristics of model impropriety of particular concern.

Instability

Let \(R(\theta) = \max_{v} \max_{h}Q(x) - \min_{v}\max_{h}Q(x) - H\log 2\).

Uninterpretability

Manageable (a.k.a. small) examples

RBMs easily are near-degenerate, unstable, and uninterpretable for large portions of parameter space.

Data coding to mitigate degeneracy

Convex hulls of the statistic space for a toy RBM with \(V = H = 1\) for \(\{0,1\}\) and \(\{-1,1\}\)-encoding enclosed by an unrestricted hull of 3-space.

Data coding to mitigate degeneracy (cont’d)

• For the \(\{-1, 1 \}\) encoding of \(\mathcal{V}\) and \(\mathcal{H}\), the origin is the center of the parameter space
• At \(\theta = 0\), the RBM is equivalent to elements of \(X\) being distributed as iid Bernoulli\(\left(\frac{1}{2}\right)\) \(\Rightarrow\) No near-degeneracy, instability, or uninterpretability!

A tale of three methods

1. Trick prior. Cancel out the normalizing term, full conditionals normally dist’d. \[ \pi(\theta) \propto \gamma(\theta)^n \exp\left(-\frac{1}{2C_{1}} \theta_{main}' \theta_{main} -\frac{1}{2C_{2}} \theta_{int}' \theta_{int}\right), \] where \(\gamma(\theta)\) is the normalizing term (Li 2014).
2. Truncated Normal prior. Use two independent truncated spherical normal distributions as priors for \(\theta_{main}\) and \(\theta_{int}\) with \(\sigma_{int} < \sigma_{main}\). Not conjugate \(\Rightarrow\) geometric adaptive MH step (Zhou 2014) and calculation of \(\gamma(\theta)\).
3. Marginalized likelihood. Marginalize out \(h\) in \(f_{\theta}(x)\), use trunc. Normal prior. \[ g_{ \theta}(v) = \sum\limits_{h \in \{-1,1\}^H} \exp\left(\sum\limits_{i = 1}^V \sum\limits_{j=1}^H \theta_{ij} v_i h_j + \sum\limits_{i = 1}^V\theta_{v_i} v_i + \sum\limits_{j = 1}^H\theta_{h_j} h_j\right) \]

Posterior distributions of images

Wrapping up

• While RBMs can be useful for classification, in the context of a statistical model as a representation of data, RBMs poor fit due to
  1. near-degeneracy,
  2. instability, and
  3. uninterpretability.
• Rigorous fitting methodology is possible but slow, and ultimately results in the empirical data distribution.