What is a transformer and how does it power LLMs?

The transformer is the neural network architecture behind all modern LLMs. Its key innovation - self-attention - lets every token in a sequence relate to every other token simultaneously, enabling parallel training on massive datasets.

AIMachine LearningLLM

Overview

Before transformers (2017, "Attention Is All You Need"), language models used RNNs that processed text sequentially - slow to train and poor at capturing long-range relationships. Transformers replaced sequential processing with **self-attention**, which lets every position in the input simultaneously relate to every other position in one operation. This parallelism is what enabled training on trillions of tokens - the foundation of modern LLMs.

Key Points

•**Self-attention**: each token "attends to" all other tokens, learning which relationships matter for predicting the next token.
•**Multi-head attention**: multiple attention heads run in parallel, each learning different types of relationships (syntax, semantics, coreference).
•**Positional encoding**: since attention has no notion of order, position information is injected as a numerical encoding.
•**Encoder vs decoder**: BERT-style models use only encoders (bidirectional, good for classification). GPT-style use only decoders (causal/autoregressive, good for generation). T5/BART use both.
•All modern LLMs (GPT-4, Claude, LLaMA, Gemini) are decoder-only transformers scaled to billions of parameters.
•The quadratic complexity of attention (O(n²) with sequence length) is why context windows have limits - and why newer architectures like Mamba explore alternatives.

Examples

Self-attention in 20 lines (NumPy pseudocode)

python

import numpy as np

def self_attention(X, W_q, W_k, W_v):
    """
    X: input matrix [seq_len × d_model]
    W_q, W_k, W_v: learned weight matrices [d_model × d_k]
    """
    d_k = W_k.shape[1]

    Q = X @ W_q  # Queries: "what am I looking for?"
    K = X @ W_k  # Keys:    "what do I contain?"
    V = X @ W_v  # Values:  "what information do I provide?"

    # Dot-product attention: how much does each query match each key?
    scores = Q @ K.T / np.sqrt(d_k)   # scale to prevent vanishing gradients

    # Softmax: normalise to probabilities (attention weights)
    weights = np.exp(scores) / np.exp(scores).sum(axis=-1, keepdims=True)

    # Weighted sum of values
    output = weights @ V   # [seq_len × d_k]
    return output, weights

# For the sentence "The cat sat":
# Token "sat" learns to attend heavily to "cat" (the subject) → grammatical agreement
# Token "cat" learns to attend to "The"  → determines the article relationship

Using a pre-trained transformer (HuggingFace)

python

from transformers import pipeline, AutoTokenizer, AutoModel
import torch

# ── Text generation (GPT-style decoder) ─────────────────────────────
generator = pipeline('text-generation', model='gpt2')
output = generator('Once upon a time, a developer wrote', max_new_tokens=50)
print(output[0]['generated_text'])

# ── Sentence embeddings (BERT-style encoder) ────────────────────────
tokenizer = AutoTokenizer.from_pretrained('sentence-transformers/all-MiniLM-L6-v2')
model = AutoModel.from_pretrained('sentence-transformers/all-MiniLM-L6-v2')

sentences = ['This is an example sentence', 'Each sentence becomes a vector']
inputs = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')

with torch.no_grad():
    outputs = model(**inputs)

# Mean pool the last hidden state to get sentence embeddings
embeddings = outputs.last_hidden_state.mean(dim=1)
print(embeddings.shape)  # torch.Size([2, 384])

Decoder-only vs encoder-only - when to use each

text

Architecture        | Models               | Best for
──────────────────────────────────────────────────────────────────
Decoder-only        | GPT-4, Claude,       | Text generation, chat,
(autoregressive)    | LLaMA, Mistral       | code generation, agents

Encoder-only        | BERT, RoBERTa,       | Classification, NER,
(bidirectional)     | all-MiniLM           | embeddings, semantic search

Encoder-decoder     | T5, BART,            | Translation, summarisation,
(seq2seq)           | mT5                  | question answering from docs

Pro Tip

You don't need to understand transformer internals to build great LLM applications - the same way you don't need to understand TCP/IP to build web apps. But knowing that attention is O(n²) explains *why* context windows cost more as they grow, and why "just increase the context window" is not a free lunch.