Large Language Models (LLMs) have revolutionized the sector of natural language processing (NLP) by demonstrating remarkable capabilities in generating human-like text, answering questions, and assisting with a big selection of language-related tasks. On the core of those powerful models lies the decoder-only transformer architecture, a variant of the unique transformer architecture proposed within the seminal paper “Attention is All You Need” by Vaswani et al.

On this comprehensive guide, we’ll explore the inner workings of decoder-based LLMs, delving into the basic constructing blocks, architectural innovations, and implementation details which have propelled these models to the forefront of NLP research and applications.

The Transformer Architecture: A Refresher

Before diving into the specifics of decoder-based LLMs, it’s essential to revisit the transformer architecture, the inspiration upon which these models are built. The transformer introduced a novel approach to sequence modeling, relying solely on attention mechanisms to capture long-range dependencies in the information, without the necessity for recurrent or convolutional layers.

The unique transformer architecture consists of two foremost components: an encoder and a decoder. The encoder processes the input sequence and generates a contextualized representation, which is then consumed by the decoder to supply the output sequence. This architecture was initially designed for machine translation tasks, where the encoder processes the input sentence within the source language, and the decoder generates the corresponding sentence within the goal language.

Self-Attention: The Key to Transformer’s Success

At the center of the transformer lies the self-attention mechanism, a robust technique that enables the model to weigh and aggregate information from different positions within the input sequence. Unlike traditional sequence models, which process input tokens sequentially, self-attention enables the model to capture dependencies between any pair of tokens, no matter their position within the sequence.

The self-attention operation could be broken down into three foremost steps:

1. Query, Key, and Value Projections: The input sequence is projected into three separate representations: queries (Q), keys (K), and values (V). These projections are obtained by multiplying the input with learned weight matrices.
2. Attention Rating Computation: For every position within the input sequence, attention scores are computed by taking the dot product between the corresponding query vector and all key vectors. These scores represent the relevance of every position to the present position being processed.
3. Weighted Sum of Values: The eye scores are normalized using a softmax function, and the resulting attention weights are used to compute a weighted sum of the worth vectors, producing the output representation for the present position.

Multi-head attention, a variant of the self-attention mechanism, allows the model to capture various kinds of relationships by computing attention scores across multiple “heads” in parallel, each with its own set of query, key, and value projections.

Architectural Variants and Configurations

While the core principles of decoder-based LLMs remain consistent, researchers have explored various architectural variants and configurations to enhance performance, efficiency, and generalization capabilities. On this section, we’ll delve into the various architectural decisions and their implications.

Architecture Types

Decoder-based LLMs could be broadly classified into three foremost types: encoder-decoder, causal decoder, and prefix decoder. Each architecture type exhibits distinct attention patterns, as illustrated in Figure 1.

Encoder-Decoder Architecture

Based on the vanilla Transformer model, the encoder-decoder architecture consists of two stacks: an encoder and a decoder. The encoder uses stacked multi-head self-attention layers to encode the input sequence and generate latent representations. The decoder then performs cross-attention on these representations to generate the goal sequence. While effective in various NLP tasks, few LLMs, equivalent to Flan-T5, adopt this architecture.

Causal Decoder Architecture

The causal decoder architecture incorporates a unidirectional attention mask, allowing each input token to attend only to past tokens and itself. Each input and output tokens are processed throughout the same decoder. Notable models like GPT-1, GPT-2, and GPT-3 are built on this architecture, with GPT-3 showcasing remarkable in-context learning capabilities. Many LLMs, including OPT, BLOOM, and Gopher, have widely adopted causal decoders.

Prefix Decoder Architecture

Also often called the non-causal decoder, the prefix decoder architecture modifies the masking mechanism of causal decoders to enable bidirectional attention over prefix tokens and unidirectional attention on generated tokens. Just like the encoder-decoder architecture, prefix decoders can encode the prefix sequence bidirectionally and predict output tokens autoregressively using shared parameters. LLMs based on prefix decoders include GLM130B and U-PaLM.

All three architecture types could be prolonged using the mixture-of-experts (MoE) scaling technique, which sparsely prompts a subset of neural network weights for every input. This approach has been employed in models like Switch Transformer and GLaM, with increasing the variety of experts or total parameter size showing significant performance improvements.

Decoder-Only Transformer: Embracing the Autoregressive Nature

While the unique transformer architecture was designed for sequence-to-sequence tasks like machine translation, many NLP tasks, equivalent to language modeling and text generation, could be framed as autoregressive problems, where the model generates one token at a time, conditioned on the previously generated tokens.

Enter the decoder-only transformer, a simplified variant of the transformer architecture that retains only the decoder component. This architecture is especially well-suited for autoregressive tasks, because it generates output tokens one after the other, leveraging the previously generated tokens as input context.

The important thing difference between the decoder-only transformer and the unique transformer decoder lies within the self-attention mechanism. Within the decoder-only setting, the self-attention operation is modified to forestall the model from attending to future tokens, a property often called causality. That is achieved through a way called “masked self-attention,” where attention scores corresponding to future positions are set to negative infinity, effectively masking them out throughout the softmax normalization step.

Architectural Components of Decoder-Based LLMs

While the core principles of self-attention and masked self-attention remain the identical, modern decoder-based LLMs have introduced several architectural innovations to enhance performance, efficiency, and generalization capabilities. Let’s explore a few of the key components and techniques employed in state-of-the-art LLMs.

Input Representation

Before processing the input sequence, decoder-based LLMs employ tokenization and embedding techniques to convert the raw text right into a numerical representation suitable for the model.

Tokenization: The tokenization process converts the input text right into a sequence of tokens, which could be words, subwords, and even individual characters, depending on the tokenization strategy employed. Popular tokenization techniques for LLMs include Byte-Pair Encoding (BPE), SentencePiece, and WordPiece. These methods aim to strike a balance between vocabulary size and representation granularity, allowing the model to handle rare or out-of-vocabulary words effectively.

Token Embeddings: After tokenization, each token is mapped to a dense vector representation called a token embedding. These embeddings are learned throughout the training process and capture semantic and syntactic relationships between tokens.

Positional Embeddings: Transformer models process the complete input sequence concurrently, lacking the inherent notion of token positions present in recurrent models. To include positional information, positional embeddings are added to the token embeddings, allowing the model to tell apart between tokens based on their positions within the sequence. Early LLMs used fixed positional embeddings based on sinusoidal functions, while newer models have explored learnable positional embeddings or alternative positional encoding techniques like rotary positional embeddings.

Multi-Head Attention Blocks

The core constructing blocks of decoder-based LLMs are multi-head attention layers, which perform the masked self-attention operation described earlier. These layers are stacked multiple times, with each layer attending to the output of the previous layer, allowing the model to capture increasingly complex dependencies and representations.

Attention Heads: Each multi-head attention layer consists of multiple “attention heads,” each with its own set of query, key, and value projections. This permits the model to take care of different elements of the input concurrently, capturing diverse relationships and patterns.

Residual Connections and Layer Normalization: To facilitate the training of deep networks and mitigate the vanishing gradient problem, decoder-based LLMs employ residual connections and layer normalization techniques. Residual connections add the input of a layer to its output, allowing gradients to flow more easily during backpropagation. Layer normalization helps to stabilize the activations and gradients, further improving training stability and performance.

Feed-Forward Layers

Along with multi-head attention layers, decoder-based LLMs incorporate feed-forward layers, which apply a straightforward feed-forward neural network to every position within the sequence. These layers introduce non-linearities and enable the model to learn more complex representations.

Activation Functions: The alternative of activation function within the feed-forward layers can significantly impact the model’s performance. While earlier LLMs relied on the widely-used ReLU activation, newer models have adopted more sophisticated activation functions just like the Gaussian Error Linear Unit (GELU) or the SwiGLU activation, which have shown improved performance.

Sparse Attention and Efficient Transformers

While the self-attention mechanism is powerful, it comes with a quadratic computational complexity with respect to the sequence length, making it computationally expensive for long sequences. To deal with this challenge, several techniques have been proposed to cut back the computational and memory requirements of self-attention, enabling efficient processing of longer sequences.

Sparse Attention: Sparse attention techniques, equivalent to the one employed within the GPT-3 model, selectively attend to a subset of positions within the input sequence, slightly than computing attention scores for all positions. This will significantly reduce the computational complexity while maintaining reasonable performance.

Sliding Window Attention: Introduced within the Mistral 7B model , sliding window attention (SWA) is a straightforward yet effective technique that restricts the eye span of every token to a hard and fast window size. This approach leverages the flexibility of transformer layers to transmit information across multiple layers, effectively increasing the eye span without the quadratic complexity of full self-attention.

Rolling Buffer Cache: To further reduce memory requirements, especially for long sequences, the Mistral 7B model employs a rolling buffer cache. This method stores and reuses the computed key and value vectors for a hard and fast window size, avoiding redundant computations and minimizing memory usage.

Grouped Query Attention: Introduced within the LLaMA 2 model, grouped query attention (GQA) is a variant of the multi-query attention mechanism that divides attention heads into groups, each group sharing a typical key and value matrix. This approach strikes a balance between the efficiency of multi-query attention and the performance of ordinary self-attention, providing improved inference times while maintaining high-quality results.