Attention Is All You Need

Problem: Most competitive sequence transduction models rely on recurrent or convolutional neural networks with encoder-decoder architectures. However, the sequential nature of recurrent networks precludes parallelization within training examples, which becomes critical at longer sequence lengths. Convolutional neural networks also face difficulties learning dependencies between distant positions as the number of required operations grows with distance.

Method: The authors propose the Transformer, a sequence transduction model based entirely on attention mechanisms that completely dispenses with recurrence and convolutions. The model uses stacked self-attention and position-wise, fully connected layers in both the encoder and decoder. Key components include Scaled Dot-Product Attention, which scales dot products by the square root of the key dimension to prevent small gradients, and Multi-Head Attention, which allows the model to jointly attend to information from different representation subspaces.

Results: On the WMT 2014 English-to-German translation task, the big Transformer model achieves a state-of-the-art score of 28.4 BLEU, outperforming previous ensembles by over 2 BLEU. On the WMT 2014 English-to-French task, the model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs. Additionally, the Transformer generalizes successfully to English constituency parsing, outperforming several task-specific models even with limited training data.

Limitations: One drawback of self-attention is the reduced effective resolution due to averaging attention-weighted positions, although this is counteracted by Multi-Head Attention. Additionally, the computational complexity of self-attention can make it less efficient when the sequence length is larger than the representation dimensionality, prompting future investigation into restricted neighborhood attention for very long sequences. Finally, using label smoothing during training improves accuracy and BLEU score but hurts model perplexity.