← Reference · Nestor G Pestelos Jr · Print this page

Machine Learning · Neural Network Architecture

Transformer Architecture

A citable reference on the decoder-only Transformer used by large language models: its three components, the attention and feed-forward internals of a block, positional encoding, and the rules that pick each next token.

Reference entry · last updated September 11, 2026 · Previous version (20260911)

1. First Principles: Sequence Modeling Without Recurrence

The Transformer architecture is a deep neural network design introduced by Vaswani et al. (2017) that models sequence dependencies entirely through self-attention operations and feed-forward projections, omitting recurrent loops and convolutional filters [1].

Prior sequence models, such as Recurrent Neural Networks (RNNs) and Long Short-Term Memory networks (LSTMs), processed tokens sequentially. That serial bottleneck prevented full hardware parallelization across training sequences. Information had to pass through sequential hidden states \(h_t\), leading to vanishing gradients across long contexts. The Transformer processes all sequence positions concurrently in training, reducing the path length for distant token interactions to \(O(1)\) operations.

Since around 2020, most large language models use one branch of this design: the decoder-only Transformer. The sections below describe that configuration.

2. The Decoder as Three Components

A decoder-only Transformer has three parts in sequence: a tokenizer, a stack of Transformer blocks, and a language modeling head [7].

• Tokenizer. Converts input text into a sequence of integer token IDs drawn from a fixed vocabulary, often near 50,000 entries. Most systems use a subword scheme such as byte-pair encoding, which grows the vocabulary by repeatedly merging the most frequent adjacent symbol pair [4]. The vocabulary is fixed before training and holds special control tokens, for example a start-of-sequence marker. A fixed vocabulary size gives every token one row in the embedding table, so the embedding lookup is a constant-time index operation. The output side is not a lookup: the language modeling head multiplies the hidden vector by a matrix of the same width to produce one logit per vocabulary entry, a cost that scales with vocabulary size and hidden width rather than an index operation (below).
• Stack of Transformer blocks. An embedding lookup maps each token ID to a dense vector of width \(d_{\text{model}}\). The vectors pass through \(N\) identical blocks. Each block applies masked multi-head self-attention and a position-wise feed-forward network, each wrapped in a residual connection and normalization (Sections 3 to 5). A block emits one vector per input position. The number of positions a forward pass can hold is the context window.
• Language modeling head. A final linear layer projects the top block's output vector at each position to \(V\) logits, one per vocabulary entry. Many models tie this projection to the input embedding matrix, which drops a large parameter block and tends to lower perplexity [5]. A softmax over the \(V\) logits gives the next-token probability distribution (Section 7).

Generation is autoregressive: the model predicts one token, appends it to the input, and runs the forward pass again. The KV cache stores the key and value projections from earlier positions so each step recomputes attention only for the new token.

3. Scaled Dot-Product Attention

The primary primitive of the Transformer is scaled dot-product attention. Given input representations packed into Query matrix \(Q\), Key matrix \(K\), and Value matrix \(V\), attention computes weighted averages of values based on query-key inner products:

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

Here \(d_k\) denotes the dimension of the keys. The scaling factor \(\frac{1}{\sqrt{d_k}}\) counteracts growth in the dot-product magnitude as dimension size expands. Without this scaling, large dot products push the softmax function into regions with tiny gradients, stalling training progress.

In causal or autoregressive decoders, an attention mask with \(-\infty\) entries is added to upper-triangular logits before the softmax step. This ensures position \(i\) cannot attend to future tokens \(j > i\).

4. Multi-Head Attention

Rather than computing a single attention distribution with dimension \(d_{\text{model}}\), multi-head attention projects queries, keys, and values \(h\) times using learned parameter matrices:

$$\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, \dots, \text{head}_h)W^O$$

$$\text{head}_i = \text{Attention}(QW_i^Q, KW_i^K, VW_i^V)$$

Projections map inputs into lower-dimensional subspaces (\(d_k = d_v = d_{\text{model}} / h\)). Multiple heads allow the model to attend simultaneously to information from different representation subspaces at different positions, capturing syntax, co-reference, and factual relations concurrently.

5. Feed-Forward Networks & Normalization

Each attention block is paired with a position-wise feed-forward network (FFN). The FFN consists of two linear transformations separated by a non-linear activation (historically ReLU, modernized to GELU or SwiGLU):

$$\text{FFN}(x) = \max(0, xW_1 + b_1)W_2 + b_2$$

Every sub-layer (attention and FFN) incorporates a residual skip connection followed by layer normalization:

$$\text{Output} = \text{LayerNorm}(x + \text{SubLayer}(x))$$

Modern implementations often adopt Pre-LayerNorm (normalizing inputs before sub-layers) or Root Mean Square Normalization (RMSNorm) for superior numerical stability during large-scale pre-training.

6. Positional Representation

Because attention operates as a permutation-equivariant set operation, the architecture possesses no built-in spatial order. Positional information must be explicitly injected into input token embeddings.

• Sinusoidal Encodings: Fixed trigonometric wave functions across dimension channels (Vaswani et al., 2017).
• Learned Absolute Embeddings: Dedicated position vectors optimized during training (Radford et al., 2018).
• Rotary Position Embedding (RoPE): Multiplicative rotation of query and key representations in complex space, encoding relative distance through inner products (Su et al., 2024) [2]. RoPE dominates contemporary frontier decoders.

7. Decoding Strategies

The language modeling head produces a probability distribution over the vocabulary at each step. Decoding is the rule that turns that distribution into the next token. With logit vector \(z\) from the head, vocabulary size \(V\), and temperature \(\tau\):

$$p(x_t = i \mid x_{\lt t}) = \frac{\exp(z_i / \tau)}{\sum_{j=1}^{V} \exp(z_j / \tau)}$$

• Greedy decoding: take the arg max of the logits at every step. It is deterministic and cheap, and tends toward repetition and generic phrasing.
• Temperature sampling: divide the logits by \(\tau\) before the softmax, then sample. Values below 1 sharpen the distribution toward its mode; values above 1 flatten it; \(\tau \to 0\) recovers greedy decoding.
• Truncated sampling: restrict sampling to a shortlist of high-probability tokens. Top-k keeps the \(k\) most probable tokens; nucleus, or top-p, sampling keeps the smallest set whose cumulative probability reaches \(p\) [6]. Both cut the low-probability tail that produces incoherent text.

8. Architectural Configurations