Meaning Space
Words become coordinates
Each token is turned into a long list of numbers, a position in a space of meaning. Words used in similar ways end up near each other, which is how a model can tell that 'invoice' and 'bill' are cousins.
Words become coordinates
Once your text is a list of numbers, each token gets turned into something richer: a position in a space of meaning. Words used in similar ways end up near each other.
Three words are about to do a lot of work, so they are worth pinning down first. A vector is just a list of numbers treated as one value. To place a point on a map you need two numbers, how far east and how far north, and that pair is a vector. Each number in the list is a dimension, which is only a formal way of saying "one of the slots". A map has two dimensions because a place needs two numbers to pin it down. An embedding is the particular vector a model uses to stand for one token.
That is the whole of the vocabulary. Everything else in this module is what happens when you give each word a position like that and then measure how far apart they are — because once meaning is a position, similarity becomes a distance you can actually calculate.
This is not a dictionary. Nobody wrote down that 'invoice' and 'receipt' are related. The positions emerged from the training task — words that appear in similar contexts drift toward similar coordinates, and the result is a map of meaning nobody drew.
Every dot is a real word at its real coordinates. Click one and watch which words come up with it.
Every dot is a real word at its real coordinates. Hover one to read it, click it and its nearest neighbours rise — the higher a word stands, the closer it is.
Like a seating plan nobody designed
Imagine a vast hall where everyone drifts toward people they have things in common with, and nobody assigns the seats. After long enough the accountants are sitting together, the musicians are across the room, and someone who does both has ended up somewhere in between. You could work out a stranger's profession from where they chose to sit down, and that position is what an embedding is.