Cascaded Text Generation with Markov Transformers

方法与贡献

用于并行文本生成的高阶 CRF 级联模型，由一个 Markov Transformer 统一参数化。

在保持结构化推断可计算的同时增强输出依赖，与扩散解码内部的联合建模相连接。

原文摘要（英文）

The two dominant approaches to neural text generation are fully autoregressive models, using serial beam search decoding, and non-autoregressive models, using parallel decoding with no output dependencies. This work proposes an autoregressive model with sub-linear parallel time generation. Noting that conditional random fields with bounded context can be decoded in parallel, we propose an efficient cascaded decoding approach for generating high-quality output. To parameterize this cascade, we introduce a Markov transformer, a variant of the popular fully autoregressive model that allows us to simultaneously decode with specific autoregressive context cutoffs. This approach requires only a small modification from standard autoregressive training, while showing competitive accuracy/speed tradeoff compared to existing methods on five machine translation datasets.

结合主题阅读

结构化解码：高阶 CRF、扩散语言模型与候选格 →

引用

Yuntian Deng, Alexander M. Rush. Cascaded Text Generation with Markov Transformers. NeurIPS 2020.

@inproceedings{deng2020cascaded,
  title = {{Cascaded Text Generation with Markov Transformers}},
  author = {Deng, Yuntian and Rush, Alexander M.},
  year = {2020},
  booktitle = {NeurIPS 2020},
  url = {https://arxiv.org/abs/2006.01112},
  eprint = {2006.01112},
  archivePrefix = {arXiv}
}