Type your own sentence and watch it transform — or use the example:
Your raw text
The cat sat on the mat.
Computers can't read words. They only understand numbers. So before we feed text to an AI, we have to convert it into numbers — one step at a time. This pipeline shows exactly how that works.
Press "Next" to begin, or click any step above to jump straight to it.
Before and after standardisation
before original
The cat sat on the mat.
after standardised
the cat sat on the mat
What changed:
First, we tidy up the text. We make everything lowercase and remove punctuation like full stops and commas. Why? Because "The" and "the" mean the same thing — we don't want the computer to think they're different words.
Think of it like washing vegetables before cooking — you just clean it up before doing the real work.
Splitting into tokens
Click any token to highlight it
Next, we split the sentence into individual tokens — usually just individual words. Each word becomes its own unit that we can process separately.
Notice that "the" appears twice! That's fine — each occurrence is its own token and will be looked up separately.
Every word gets a number (an index)
token
index in vocabulary
note
The AI has a big list of words called a vocabulary — like a dictionary. Each word has a position number (its index). We look up each token and replace it with that number.
That's why "the" gets the same number both times — it's the same word in the dictionary!
One-hot encoding — turning an index into a vector
Click a token to see its one-hot vector:
Select a token above
A one-hot vector is a list of zeros with exactly one "1" — at the position matching the word's index. The vocabulary has thousands of words, so the vector is very long, with one "1" and the rest "0".
We only show the first 8 positions here to keep it simple. In real life the vector might have 50,000 positions!