An evaluation of the intuition behind the notion of Key, Query, and Value in Transformer architecture and why is it used.

Recent years have seen the Transformer architecture make waves in the sphere of natural language processing (NLP), achieving state-of-the-art leads to a wide range of tasks including machine translation, language modeling, and text summarization, in addition to other domains of AI i.e. Vision, Speech, RL, etc.

Vaswani et al. (2017), first introduced the transformer of their paper “Attention Is All You Need”, during which they used the self-attention mechanism without incorporating recurrent connections while the model can focus selectively on specific portions of input sequences.

Specifically, previous sequence models, reminiscent of recurrent encoder-decoder models, were limited of their ability to capture long-term dependencies and parallel computations. In truth, right before the Transformers paper got here out in 2017, state-of-the-art performance in most NLP tasks was obtained through the use of RNNs with an attention mechanism on top, so attention form of existed before transformers. By introducing the multi-head attention mechanism by itself, and dropping the RNN part, the transformer architecture resolves these issues by allowing multiple independent attention mechanisms.

On this post, we are going to go over certainly one of the small print of this architecture, namely the Query, Key, and Values, and take a look at to make sense of the intuition used behind this part.

Note that this post assumes you’re already accustomed to some basic concepts in NLP and deep learning reminiscent of embeddings, Linear (dense) layers, and on the whole how an easy neural network works.

First, let’s start understanding what the eye mechanism is trying to realize. And for the sake of simplicity, let’s start with an easy case of sequential data to know what problem exactly…