Bayesian Attention Modules

Method and contribution

Builds distributions over attention weights from normalized reparameterizable variables and learns them with Bayesian regularization.

Extends probabilistic attention to modular stochastic weights with data-dependent priors.

Read in context

Attention as a Latent Variable: Modeling a Distribution over Attention →

Citation

Xinjie Fan, Shujian Zhang, Bo Chen, Mingyuan Zhou. Bayesian Attention Modules. NeurIPS 2020.

@inproceedings{fan2020bayesian,
  title = {{Bayesian Attention Modules}},
  author = {Fan, Xinjie and Zhang, Shujian and Chen, Bo and Zhou, Mingyuan},
  year = {2020},
  booktitle = {NeurIPS 2020},
  url = {https://arxiv.org/abs/2010.10604},
  eprint = {2010.10604},
  archivePrefix = {arXiv}
}