Energy-based text generation: residual models, resampling and diffusion

The residual text-generation model

Yuntian Deng, Anton Bakhtin, Myle Ott, Arthur Szlam and Marc’Aurelio Ranzato introduced the ICLR 2020 text-generation framework. Its arXiv version is dated 22 April 2020. A fixed autoregressive LM supplies candidate sequences; a bidirectional energy function scores them. The paper develops importance-weighted conditional generation and lower and upper log-probability bounds for perplexity evaluation.

The distribution is p(y | x) = q(y | x) exp(−E(x,y)) / Z(x). Here x is the context, y a complete continuation, q the base model, E the residual energy, and Z the normalizer over continuations. Lower energy increases probability relative to q. Noise-contrastive estimation trains the energy to distinguish real text from base-model samples.

Sources: Residual Energy-Based Models for Text Generation

Why a discriminator can change a generator

The JMLR 2021 article, Residual Energy-Based Models for Text, develops the connection between distinguishing generated text and improving its distribution. Anton Bakhtin and Yuntian Deng contributed equally; Sam Gross, Myle Ott, Marc’Aurelio Ranzato and Arthur Szlam complete the author list. A discriminator supplies information about where a generator overproduces or underproduces text. A residual model turns that information into a probability correction.

Read the product as two separate jobs. The base model provides fluent proposals and their relative frequency. The energy changes their relative desirability after viewing the sequence. An energy of zero everywhere leaves the base distribution unchanged. Adding the same constant to every energy also leaves the normalized distribution unchanged, because the constant cancels between numerator and denominator.

This gives a practical debugging check: changing a shared energy offset must not change candidate-selection probabilities. Changing a single candidate’s energy must change its probability in the expected direction. Those are algebraic properties, independent of which neural network computes the scores.

Sources: Residual Energy-Based Models for Text

Sample, score, then resample

Draw candidates from q, score each with E, and select with normalized weights proportional to exp(−E). This is importance resampling. Taking the highest-scoring candidate instead is reranking: it changes the selection rule. Both use a proposal-and-scoring pipeline, but define different output distributions.

For a worked example, assign three sampled candidates energies 0, log(2), and log(4). Their weights are 1, 1/2, and 1/4, giving selection probabilities 4/7, 2/7, and 1/7. Maximum-score selection always chooses the first candidate. Resampling preserves a distribution across the pool.

The base probability is already represented by drawing the pool from q. Multiplying each pool weight by q again changes the intended importance calculation. Likewise, a finite-pool normalizer is not an exact sum over all possible text. Increasing the number of proposals changes the approximation and the compute budget; changing the energy changes the target distribution.

Sources: Residual Energy-Based Models for Text GenerationResidual Energy-Based Models for Text

The direct connection to diffusion language models

Energy-Based Diffusion Language Models (EDLM), by Minkai Xu and colleagues, was released on 28 October 2024 and published at ICLR 2025. It multiplies a diffusion denoiser’s distribution by a sequence-level residual energy conditioned on the noisy sequence and timestep. This corrects dependencies missed by factorized token predictions within a denoising step.

EDLM uses either an autoregressive-model-derived energy or an energy trained by NCE, then generates through parallel importance sampling. It cites Deng et al. (2020) in its related work and explicitly uses that work in partition-function estimation. The relationship is mathematical and algorithmic: residual correction, contrastive training, resampling and likelihood estimation.

The 2020 model corrects an autoregressive proposal for text generation. EDLM corrects a denoising distribution inside an iterative diffusion process. The location of the correction changes; the base-times-energy construction connects them.

Sources: Residual Energy-Based Models for Text GenerationEnergy-Based Diffusion Language Models for Text Generation

Compare the base, energy and inference

Wang and Ou’s neural TRF paper supplies an earlier residual formulation and NCE approach, credited explicitly in the 2020 paper. The 2020 contribution develops conditional text generation and likelihood evaluation around the residual model. Unified Energy (2026) extends the diffusion discussion by analyzing dependency and invariance errors and combining their energy corrections.

Sources: Learning Neural Trans-Dimensional Random Field Language Models with Noise-Contrastive EstimationResidual Energy-Based Models for Text GenerationEnergy-Based Diffusion Language Models for Text GenerationUnified Energy for Invariant and Independent Decoding in Diffusion Language Models

Use the energy example to inspect the mechanism

Change the energy strength β over the three-outcome distribution in the example. At β = 0, the probabilities are the base distribution. Increasing β shifts mass toward lower-energy outcomes, while the base probabilities still affect the result. The example computes the exact normalized distribution over three possible sequences; it does not draw a candidate pool or call a language model.

For importance resampling from candidates already drawn from the base model, the pool weights are exp(−E(x)). The base-model factor is already accounted for by that sampling step. This distinction connects the normalized model to its practical sampling procedure.

Papers & implementations