Nicolas Chopin

Noise contrastive estimation: asymptotics, comparison with MC-MLE

A statistical model is said to be un-normalised when its likelihood
function involves an intractable normalising constant. Two popular
methods for parameter inference for these models are MC-MLE (Monte
Carlo maximum likelihood estimation), and NCE (noise contrastive
estimation); both methods rely on simulating artificial data-points to
approximate the normalising constant. While the asymptotics of MC-MLE
have been established under general hypotheses (Geyer, 1994), this is
not so for NCE. We establish consistency and asymptotic normality of
NCE estimators under mild assumptions. We compare NCE and MC-MLE under
several asymptotic regimes. In particular, we show that, when m goes
to infinity while n is fixed (m and n being respectively the number of
artificial data-points, and actual data-points), the two estimators
are asymptotically equivalent. Conversely, we prove that, when the
artificial data-points are IID, and when n goes to infinity while m/n
converges to a positive constant, the asymptotic variance of a NCE
estimator is always smaller than the asymptotic variance of the
corresponding MC-MLE estimator. We illustrate the variance reduction
brought by NCE through a numerical study. (based on joint work with Simon Barthelmé and Lionel Riou-Durand)