{
  "id": "energy-based-text-generation",
  "title": "Energy-based text generation: residual models, resampling and diffusion",
  "eyebrow": "Global sequence models",
  "description": "Residual Energy-Based Models for Text Generation connects sequence-level energy, noise-contrastive estimation and importance resampling to energy-based diffusion language models.",
  "lead": "Residual Energy-Based Models for Text Generation (ICLR 2020) turns a pretrained language model and a whole-sequence energy into a globally normalized generator. Energy-Based Diffusion Language Models (ICLR 2025) explicitly connects to this work and applies residual energy correction inside diffusion denoising. The shared mechanism is a base distribution multiplied by a learned preference over complete candidate sequences.",
  "original_papers": [
    "residual-ebm",
    "residual-ebm-jmlr"
  ],
  "terms": [
    "energy-based text generation",
    "globally normalized language models",
    "sample and rerank",
    "importance resampling",
    "energy-based diffusion language models",
    "sequence-level energy",
    "dependency-aware parallel decoding"
  ],
  "demo": "energy",
  "related_guides": [
    "structured-decoding",
    "probabilistic-attention"
  ],
  "reading_order": [
    "residual-ebm",
    "residual-ebm-jmlr",
    "edlm",
    "unified-energy",
    "neural-trf-nce"
  ],
  "sections": [
    {
      "id": "original-method",
      "title": "The residual text-generation model",
      "paragraphs": [
        "Yuntian Deng, Anton Bakhtin, Myle Ott, Arthur Szlam and Marc’Aurelio Ranzato introduced the ICLR 2020 text-generation framework. Its arXiv version is dated 22 April 2020. A fixed autoregressive LM supplies candidate sequences; a bidirectional energy function scores them. The paper develops importance-weighted conditional generation and lower and upper log-probability bounds for perplexity evaluation.",
        "The distribution is <code>p(y | x) = q(y | x) exp(−E(x,y)) / Z(x)</code>. Here x is the context, y a complete continuation, q the base model, E the residual energy, and Z the normalizer over continuations. Lower energy increases probability relative to q. Noise-contrastive estimation trains the energy to distinguish real text from base-model samples."
      ],
      "citations": [
        "residual-ebm"
      ]
    },
    {
      "id": "meaning-of-residual",
      "title": "Why a discriminator can change a generator",
      "paragraphs": [
        "The JMLR 2021 article, Residual Energy-Based Models for Text, develops the connection between distinguishing generated text and improving its distribution. Anton Bakhtin and Yuntian Deng contributed equally; Sam Gross, Myle Ott, Marc’Aurelio Ranzato and Arthur Szlam complete the author list. A discriminator supplies information about where a generator overproduces or underproduces text. A residual model turns that information into a probability correction.",
        "Read the product as two separate jobs. The base model provides fluent proposals and their relative frequency. The energy changes their relative desirability after viewing the sequence. An energy of zero everywhere leaves the base distribution unchanged. Adding the same constant to every energy also leaves the normalized distribution unchanged, because the constant cancels between numerator and denominator.",
        "This gives a practical debugging check: changing a shared energy offset must not change candidate-selection probabilities. Changing a single candidate’s energy must change its probability in the expected direction. Those are algebraic properties, independent of which neural network computes the scores."
      ],
      "citations": [
        "residual-ebm-jmlr"
      ]
    },
    {
      "id": "sample-score-resample",
      "title": "Sample, score, then resample",
      "paragraphs": [
        "Draw candidates from q, score each with E, and select with normalized weights proportional to <code>exp(−E)</code>. This is importance resampling. Taking the highest-scoring candidate instead is reranking: it changes the selection rule. Both use a proposal-and-scoring pipeline, but define different output distributions.",
        "For a worked example, assign three sampled candidates energies 0, log(2), and log(4). Their weights are 1, 1/2, and 1/4, giving selection probabilities 4/7, 2/7, and 1/7. Maximum-score selection always chooses the first candidate. Resampling preserves a distribution across the pool.",
        "The base probability is already represented by drawing the pool from q. Multiplying each pool weight by q again changes the intended importance calculation. Likewise, a finite-pool normalizer is not an exact sum over all possible text. Increasing the number of proposals changes the approximation and the compute budget; changing the energy changes the target distribution."
      ],
      "citations": [
        "residual-ebm",
        "residual-ebm-jmlr"
      ]
    },
    {
      "id": "diffusion-connection",
      "title": "The direct connection to diffusion language models",
      "paragraphs": [
        "Energy-Based Diffusion Language Models (EDLM), by Minkai Xu and colleagues, was released on 28 October 2024 and published at ICLR 2025. It multiplies a diffusion denoiser’s distribution by a sequence-level residual energy conditioned on the noisy sequence and timestep. This corrects dependencies missed by factorized token predictions within a denoising step.",
        "EDLM uses either an autoregressive-model-derived energy or an energy trained by NCE, then generates through parallel importance sampling. It cites Deng et al. (2020) in its related work and explicitly uses that work in partition-function estimation. The relationship is mathematical and algorithmic: residual correction, contrastive training, resampling and likelihood estimation.",
        "The 2020 model corrects an autoregressive proposal for text generation. EDLM corrects a denoising distribution inside an iterative diffusion process. The location of the correction changes; the base-times-energy construction connects them."
      ],
      "citations": [
        "residual-ebm",
        "edlm"
      ]
    },
    {
      "id": "method-comparison",
      "title": "Compare the base, energy and inference",
      "table": {
        "columns": [
          "Method",
          "Base and correction",
          "Inference or evaluation focus"
        ],
        "rows": [
          [
            "Neural TRF with NCE · 2018",
            "Reference LM with exponential tilting",
            "Speech-recognition rescoring"
          ],
          [
            "Residual EBM · 2020",
            "Autoregressive LM × sequence energy",
            "Conditional generation by importance resampling; likelihood bounds"
          ],
          [
            "EDLM · 2025",
            "Diffusion denoiser × conditional sequence energy",
            "Parallel proposals and resampling within denoising"
          ],
          [
            "Unified Energy · 2026",
            "Energy corrections for dependency and invariance",
            "Improved diffusion decoding"
          ]
        ]
      },
      "paragraphs": [
        "Wang and Ou’s neural TRF paper supplies an earlier residual formulation and NCE approach, credited explicitly in the 2020 paper. The 2020 contribution develops conditional text generation and likelihood evaluation around the residual model. Unified Energy (2026) extends the diffusion discussion by analyzing dependency and invariance errors and combining their energy corrections."
      ],
      "citations": [
        "neural-trf-nce",
        "residual-ebm",
        "edlm",
        "unified-energy"
      ]
    },
    {
      "id": "reading-the-demo",
      "title": "Use the energy example to inspect the mechanism",
      "paragraphs": [
        "Change the energy strength β over the three-outcome distribution in the example. At β = 0, the probabilities are the base distribution. Increasing β shifts mass toward lower-energy outcomes, while the base probabilities still affect the result. The example computes the exact normalized distribution over three possible sequences; it does not draw a candidate pool or call a language model.",
        "For importance resampling from candidates already drawn from the base model, the pool weights are exp(−E(x)). The base-model factor is already accounted for by that sampling step. This distinction connects the normalized model to its practical sampling procedure."
      ]
    }
  ],
  "updated_at": "2026-11-10",
  "url": "https://yuntiandeng.com/research/energy-based-text-generation/",
  "papers": [
    {
      "id": "residual-ebm",
      "title": "Residual Energy-Based Models for Text Generation",
      "authors": [
        {
          "given": "Yuntian",
          "family": "Deng"
        },
        {
          "given": "Anton",
          "family": "Bakhtin"
        },
        {
          "given": "Myle",
          "family": "Ott"
        },
        {
          "given": "Arthur",
          "family": "Szlam"
        },
        {
          "given": "Marc’Aurelio",
          "family": "Ranzato"
        }
      ],
      "year": 2020,
      "venue": "ICLR 2020",
      "paper_url": "https://arxiv.org/html/2004.11714v1",
      "summary": "Combines a fixed autoregressive language model with a learned sequence-level energy, NCE training, conditional generation by importance resampling, and likelihood bounds.",
      "connection": "Original text-generation and evaluation framework connecting sequence-level residual energies to modern energy-corrected diffusion decoding.",
      "bibtex_type": "inproceedings",
      "citation_key": "residualebm",
      "source_urls": [
        "https://arxiv.org/html/2004.11714v1"
      ],
      "pdf_url": "https://arxiv.org/pdf/2004.11714",
      "code_url": "https://github.com/da03/Residual-EBM",
      "arxiv_id": "2004.11714",
      "arxiv_first_posted": "2020-04-22",
      "abstract": "Text generation is ubiquitous in many NLP tasks, from summarization, to dialogue and machine translation. The dominant parametric approach is based on locally normalized models which predict one word at a time. While these work remarkably well, they are plagued by exposure bias due to the greedy nature of the generation process. In this work, we investigate un-normalized energy-based models (EBMs) which operate not at the token but at the sequence level. In order to make training tractable, we first work in the residual of a pretrained locally normalized language model and second we train using noise contrastive estimation. Furthermore, since the EBM works at the sequence level, we can leverage pretrained bi-directional contextual representations, such as BERT and RoBERTa. Our experiments on two large language modeling datasets show that residual EBMs yield lower perplexity compared to locally normalized baselines. Moreover, generation via importance sampling is very efficient and of higher quality than the baseline models according to human evaluation.",
      "abstract_source": "https://arxiv.org/html/2004.11714v1",
      "updated_at": "2026-11-10"
    },
    {
      "id": "residual-ebm-jmlr",
      "title": "Residual Energy-Based Models for Text",
      "authors": [
        {
          "given": "Anton",
          "family": "Bakhtin"
        },
        {
          "given": "Yuntian",
          "family": "Deng"
        },
        {
          "given": "Sam",
          "family": "Gross"
        },
        {
          "given": "Myle",
          "family": "Ott"
        },
        {
          "given": "Marc’Aurelio",
          "family": "Ranzato"
        },
        {
          "given": "Arthur",
          "family": "Szlam"
        }
      ],
      "year": 2021,
      "venue": "JMLR 22(40):1–41",
      "paper_url": "https://www.jmlr.org/papers/v22/20-326.html",
      "summary": "Studies statistical discrimination of human and model text and incorporates learned discriminators into globally normalized generation.",
      "connection": "Extended treatment of the discriminator-to-generator connection and residual energy modeling.",
      "bibtex_type": "article",
      "citation_key": "residualebmjmlr",
      "source_urls": [
        "https://www.jmlr.org/papers/v22/20-326.html"
      ],
      "pdf_url": "https://www.jmlr.org/papers/volume22/20-326/20-326.pdf",
      "equal_contribution": [
        "Anton Bakhtin",
        "Yuntian Deng"
      ],
      "container_title": "Journal of Machine Learning Research",
      "volume": "22",
      "issue": "40",
      "pages": "1–41",
      "abstract": "Current large-scale auto-regressive language models display impressive fluency and can generate convincing text. In this work we start by asking the question: Can the generations of these models be reliably distinguished from real text by statistical discriminators? We find experimentally that the answer is affirmative when we have access to the training data for the model, and guardedly affirmative even if we do not. This suggests that the auto-regressive models can be improved by incorporating the (globally normalized) discriminators into the generative process. We give a formalism for this using the Energy-Based Model framework, and show that it indeed improves the results of the generative models, measured both in terms of perplexity and in terms of human evaluation.",
      "abstract_source": "https://www.jmlr.org/papers/v22/20-326.html",
      "updated_at": "2026-11-10"
    },
    {
      "id": "edlm",
      "title": "Energy-Based Diffusion Language Models for Text Generation",
      "authors": [
        {
          "given": "Minkai",
          "family": "Xu"
        },
        {
          "given": "Tomas",
          "family": "Geffner"
        },
        {
          "given": "Karsten",
          "family": "Kreis"
        },
        {
          "given": "Weili",
          "family": "Nie"
        },
        {
          "given": "Yilun",
          "family": "Xu"
        },
        {
          "given": "Jure",
          "family": "Leskovec"
        },
        {
          "given": "Stefano",
          "family": "Ermon"
        },
        {
          "given": "Arash",
          "family": "Vahdat"
        }
      ],
      "year": 2025,
      "venue": "ICLR 2025",
      "paper_url": "https://arxiv.org/html/2410.21357v4",
      "summary": "Adds a full-sequence residual energy to each diffusion denoising distribution, with autoregressive or NCE parameterizations and parallel importance sampling.",
      "connection": "Directly cites Deng et al. (2020), carrying residual correction and partition estimation into diffusion language modeling.",
      "bibtex_type": "inproceedings",
      "citation_key": "edlm",
      "source_urls": [
        "https://arxiv.org/html/2410.21357v4"
      ],
      "arxiv_id": "2410.21357",
      "arxiv_first_posted": "2024-10-28",
      "updated_at": "2026-11-10"
    },
    {
      "id": "unified-energy",
      "title": "Unified Energy for Invariant and Independent Decoding in Diffusion Language Models",
      "authors": [
        {
          "given": "Yuchen",
          "family": "Yan"
        },
        {
          "given": "Minkai",
          "family": "Xu"
        },
        {
          "given": "Zaiquan",
          "family": "Yang"
        },
        {
          "given": "Yatao",
          "family": "Bian"
        }
      ],
      "year": 2026,
      "venue": "arXiv:2606.09159",
      "paper_url": "https://arxiv.org/html/2606.09159v1",
      "summary": "Combines energy corrections for dependency and invariance in diffusion decoding.",
      "connection": "Extends the contemporary energy-corrected diffusion direction with a unified treatment of decoding errors.",
      "bibtex_type": "article",
      "citation_key": "unifiedenergy",
      "source_urls": [
        "https://arxiv.org/html/2606.09159v1"
      ],
      "arxiv_id": "2606.09159",
      "arxiv_first_posted": "2026-06-08",
      "publication_type": "preprint",
      "container_title": "arXiv",
      "updated_at": "2026-11-10"
    },
    {
      "id": "neural-trf-nce",
      "title": "Learning Neural Trans-Dimensional Random Field Language Models with Noise-Contrastive Estimation",
      "authors": [
        {
          "given": "Bin",
          "family": "Wang"
        },
        {
          "given": "Zhijian",
          "family": "Ou"
        }
      ],
      "year": 2018,
      "venue": "ICASSP 2018",
      "paper_url": "https://arxiv.org/abs/1710.10739",
      "summary": "Uses an exponentially tilted reference distribution and noise-contrastive estimation for neural random-field language models.",
      "connection": "Earlier residual formulation and NCE training explicitly discussed in the 2020 paper; its evaluation centers on speech-recognition rescoring.",
      "bibtex_type": "inproceedings",
      "citation_key": "neuraltrfnce",
      "source_urls": [
        "https://arxiv.org/abs/1710.10739"
      ],
      "arxiv_id": "1710.10739",
      "arxiv_first_posted": "2017-10-30",
      "updated_at": "2026-11-10"
    }
  ]
}