I built an LLM from scratch: what nobody tells you about transformers

This piece is published as an outline: the statement and the structure are settled, the full text is coming.

Why bother

tinyllm: a Llama-style decoder-only model, written to understand transformers from the inside rather than through somebody else’s API.

Attention, derived

What the scaled dot product is really computing, and why dividing by the square root of d_k is a consequence rather than a trick.

What tutorials skip

Rotary positional embeddings, pre-normalisation, and why the order of operations decides whether training converges at all.

How it changed the way I use large models

Understanding the mechanism changes your expectations: you stop asking the model for things its architecture cannot give you.