Attention
The mechanism inside a transformer that lets the model weigh how much each word should influence every other word when forming meaning.
It is how a model keeps track of context and relationships across a sentence or a document.
Frequently asked questions
How does an AI keep track of what a sentence is about?
Through a mechanism called attention, which lets the model weigh how much each word should influence every other word when forming meaning. That is how it connects a pronoun back to the right noun or keeps a long thought coherent.
How does attention relate to the transformer?
Attention is the core mechanism that lives inside a transformer and does much of its work. The transformer is the overall design, and attention is the part that decides which words matter most to each other.
Why does attention matter for long documents?
It is how a model tracks relationships across a whole passage, linking ideas that sit far apart. Strong attention is part of why a model can follow context over a long document rather than losing the thread.
New to all this? Start with what an AI agent really is, browse the full glossary, or explore the learning hub.
← Back to the glossary