What is Transformer?

How it works

The transformer was introduced by Google researchers in the 2017 paper "Attention Is All You Need". Text is split into tokens, and each token becomes a vector (an embedding) with information about its position added. The model then stacks many identical layers, each with two main parts:

• Self-attention. Every token computes how relevant every other token is to it and pulls in information accordingly. In "The trophy did not fit in the case because it was too big", attention is how the model links "it" to "trophy". Several attention "heads" run side by side, each learning different relationships.
• A feed-forward network that transforms each token's vector on its own. In mixture-of-experts models this part is split into many experts with a router.

The original design had an encoder and a decoder for translation. Most chat LLMs use only the decoder half, with "causal" attention so each token can only look at earlier tokens, and generate text one token at a time.

Why it matters

• Parallel training. Older recurrent networks read text one word after another. Transformers process all positions at once, which suits GPUs and made training on internet-scale data practical.
• It scales. Bigger transformers trained on more data have kept getting better, which is the basis of the GPT, Claude, Gemini, Llama and DeepSeek families.
• It is general. The same architecture handles images (split into patches), audio and video, which is why it underpins multimodal AI.
• Its cost grows with context. Standard self-attention compares every token with every other, so compute and memory rise steeply as the context window grows. Much current research targets that cost.

Example

When you ask a model to fix a bug in a 300-line file, attention lets the token where the error occurs draw on the function definition 200 lines earlier. That long-range link is what transformers do well and earlier architectures struggled with.