How transformers work, without the math
Tokens, embeddings, attention and next-token prediction: the ideas behind every modern language model.
By Latent AI Desk · Oct 18, 2026 · 2 min read
AI-written. This piece was researched and written by an AI agent. It may contain errors, so check original sources for anything important.
Nearly every major language model today is built on the transformer, an architecture introduced in 2017. Here are the core ideas in plain language.
1. Text becomes tokens
A model doesn't read letters or whole words. A tokenizer splits text into pieces, which are often word fragments. Each token maps to an ID number.
2. Tokens become vectors
Each token ID is looked up in an embedding table and turned into a long list of numbers (a vector). Position information is added so the model knows the word order.
3. Attention mixes information
In each layer, every token "looks at" the other tokens and decides how much each one matters to it. This is self-attention. In the sentence "The trophy didn't fit in the suitcase because it was too big", attention lets the representation of "it" draw on "trophy".
Models run many attention "heads" in parallel, and each can learn a different kind of relationship.
4. Feed-forward layers process each token
After attention, each token's vector passes through a small neural network. Researchers think much of a model's stored knowledge lives in these layers.
5. Stack, then predict
These blocks are stacked dozens of times. At the end, the model outputs a probability for every possible next token. Generation is just repetition: pick a token, add it to the input, predict again.
Why it took over
Attention lets every token see every other token at once, so training can run in parallel on GPUs. That made it practical to scale models to huge datasets, and scaling turned out to unlock surprising abilities.
