Page "Gibbs sampling" Paragraph 39

* A collapsed Gibbs sampler integrates out ( marginalizes over ) one or more variables when sampling for some other variable.

For example, imagine that a model consists of three variables A, B, and C. A simple Gibbs sampler would sample from p ( A | B, C ), then p ( B | A, C ), then p ( C | A, B ).

A collapsed Gibbs sampler might replace the sampling step for A with a sample taken from the marginal distribution p ( A | C ), with variable B integrated out in this case.

Alternatively, variable B could be collapsed out entirely, alternately sampling from p ( A | C ) and p ( C | A ) and not sampling over B at all.

The distribution over a variable A that arises when collapsing a parent variable B is called a compound distribution ; sampling from this distribution is generally tractable when B is the conjugate prior for A, particularly when A and B are members of the exponential family.

For more information, see the article on compound distributions or Liu ( 1994 ).