Recipes from Papaspiliopoulos, Roberts, Sköld (2007). There is a pretty large prior literature here from before the invention of performant modern PPLs like Stan or Pyro. The current userbase of heirarchical Bayesian models specified in PPLs – I'm thinking mostly of data scientists in industry – is pretty separate from the then-users of BUGS / JAGS / WINBUGS, so I think it might be worth highlighting some of these results in the context of HMC / industry.
Non-centred parameterizations are a common tool for the Bayesian modeller, see here, (link to my own Neal's funnel) page, probably want a third link. It seems like these are originally specified in the context of blocked Gibbs samplers.
These are most often deployed when using location-scale families in heirarchical modelling. But the universe of possible model is large –- often we would like an equivalent parameterization for a model which is not location-scale. The common location scale reparameterization is a nice special case of a general strategy, in which one samples from some "nice" generator distribution, pushing the samples through a map to transform them into the quantity of interest, such that the generator in uncorellated apriori from the rest of parameter space.
One way to think about this is that, for any set of sampler-visible parameters , somewhere in the world there exists such that transports a standard Normal to the correct posterior marginal (see my blog on Doob's h-transform, which I haven't written yet). Sampling from a standard Normal induces an optimal distribution for HMC, so we'd like to move to that world when possible.
Maybe we can't find such a function. But we can do the easier job of finding alternate coordinates (or auxillary variable augementations) that bring us closer to that world.
This paper exists in the following setting:
We have data , parameters of interest , latent parameters and hierarchical model:
we're interested in the family of possible latent parameterization such that, after marginalizing out , the posterior is unchanged:
A parameterization here is some construction where we sample such that, for some function , we can construct .