Attention
What the model looks at
To work out what a word means here, the model weighs every other word in the passage and pulls hardest on the ones that matter. That weighing is attention, and it is the mechanism the whole transformer is named after.
Words do not have fixed meanings
Giving each word a position in meaning space is not enough, because words change meaning depending on what is around them. Attention is the mechanism that handles it.
Consider a word like 'bank', which on its own is ambiguous. In 'the bank approved the loan' it is a financial institution, and in 'the bank of the river' it is the edge of a waterway. The word on the page is identical in both, so everything about what it means has come from its neighbours.
So before the model predicts anything, every word looks at every other word and rebuilds itself as a blend of them. A word's representation stops being a fixed point and becomes something assembled fresh from its context, every time.
The same token in two sentences. Click it in each and watch which neighbours it leans on.
Click any word. Its strongest link is the one drawn in colour, and how high it stands is how hard it lands.
“bank” is one token in both sentences. Click it in each and watch which neighbours it leans on.
Like working out who “they” means at a dinner party
Someone says “they were furious about it.” To understand that sentence you glance back over the last few minutes of conversation, weighting recent and relevant parts heavily and ignoring most of the rest. You are not applying a rule. You are working out which earlier words this one depends on. That weighting is attention.
In “the trophy did not fit in the suitcase because it was too big”, what does “it” refer to?