Attention

AI concepts
About 1 min read

A mechanism that assigns higher weights to the information most important to the current task among all elements in the input sequence. It overcomes the limitations of fixed vector sizes and precisely extracts variable contexts to enhance the AI's understanding.

Also known as
AttentionAttention MechanismSelf-AttentionSelf-Attention

Detailed explanation

Attention is a deep learning technique that processes information by focusing on specific parts of the data. It determines the weights to be assigned to 'Values' by calculating the similarity between 'Queries' and 'Keys', allowing the model to accurately capture correlations between distant elements in a sequence. It resolved the bottleneck and long-term dependency issues of traditional RNNs, and evolved into 'Self-Attention'—which processes an entire sentence at once—becoming the core architecture of the Transformer and modern Large Language Models (LLMs).

Why it matters in tool selection

The efficiency of the attention mechanism determines the size of the AI tool's 'Context Window' and its processing speed. Modern AI tools adopt technologies like FlashAttention and Sparse Attention to drastically reduce computational costs while maintaining accuracy even when analyzing massive documents.

What to check

  • Does information loss or distortion (hallucination) occur when processing long contexts?
  • Is the response speed optimized by supporting latest acceleration technologies such as FlashAttention?
  • Can it visualize or explain how weights are assigned to specific parts of the input data?

Example

In the English sentence 'The animal didn't cross the street because it was too tired', the attention mechanism accurately identifies through contextual weight calculation that 'it' refers to the 'animal', not the 'street'.

Confusing terms

RNN (Recurrent Neural Network)

Receives data sequentially and is prone to forgetting earlier information, whereas attention references the entire data in parallel all at once.

Self-Attention

A type of attention, which is a special method of calculating relationships between words within the input sentence.

Related terms

TransformerContext Window