Type a sentence and watch it transform step by step into numbers the AI can understand:
step 1 your raw text
The cat sat on the mat
Computers only understand numbers. So before an AI can read your sentence, we need to convert every word into a list of numbers. This pipeline does exactly that — one step at a time.
Press "Next" to start, or click any step tab above to jump straight to it.
step 2 cleaning up the text
before
The cat sat on the mat.
after
the cat sat on the mat
Changes made: lowercased · removed punctuation
First, we tidy things up. We make everything lowercase and remove full stops and commas. This is called standardisation.
Why? Because "Cat" and "cat" mean the same thing. We don't want the computer to think they're two different words just because of a capital letter.
step 3 splitting into tokens
Each word becomes its own token:
We chop the sentence into individual words. Each word is now a separate unit called a token — like cutting a sausage into slices.
Notice: "the" appears twice, and that's fine! Each one is its own token and gets processed separately.
step 4 looking up word embeddings
Click a word to see its embedding — a short list of numbers that captures its meaning:
Instead of one-hot encoding (a huge list of mostly zeros), embeddings give each word a short, meaningful list of numbers — typically 100 to 300 values. These numbers were learned by the AI from reading millions of sentences.
Similar words (like "cat" and "kitten") end up with similar numbers. The AI can use this similarity to understand language!
step 5 the word map — similar words live near each other
Click any word to find its closest neighbours:
Click a word on the map above.
Because embeddings capture meaning, similar words end up close together on the map. Animals cluster near each other. Verbs cluster together. This is why an AI that learns about "cats" also partly understands "kittens" — they live in the same neighbourhood!