Exploring the power of Attention mechanism in deep Learning

1. Exploring the power of Attention mechanism in deep Learning leewayhertz.com/attention-mechanism November 9, 2023 In an era where machines are increasingly expected to comprehend, generate, and interact with human language and sensory data, the concept of “attention” has emerged as a guiding beacon in the realm of artificial intelligence. Imagine a model’s ability to focus on the most relevant details while processing vast amounts of information—a capacity mirroring the selective awareness that defines human cognition. Welcome to the world of attention models, a pivotal breakthrough in the field of machine learning that is significantly changing how we approach tasks like natural language understanding, image analysis, and more. Ever since the 2017 release of the influential paper, Attention Is All You Need, “attention networks” have been a talking point in the tech sphere and have found extensive application across various domains, primarily in Natural Language Processing (NLP) and Computer Vision. The concept of Deep Learning (DL) that was purely theoretical less than two decades ago is now being utilized to address tangible problems like converting speech into text transcripts and performing complex image recognition tasks, transforming fields such as computer vision and natural language processing. The core enabler behind these applications is a concept known as the attention mechanism or attention model. 1/27

2. To delve deeper, we must first understand what ‘attention’ implies and how it operates within the context of DL. We will also explore related terms such as encoder, and decoder. This article offers a comprehensive overview of attention mechanism and related concepts and provides a clear understanding of how these technologies are implemented in real-world scenarios. The introduction of attention mechanism in deep learning How does seq2seq model work? Limitations of the context vector Introducing attention mechanism: The solution Types of attention mechanism Bahdanau attention/ Additive attention Luong attention/ Global attention/ Multiplicative attention Self-attention Multi-head attention How does attention mechanism work? Self-attention mechanism Multi-head attention mechanism Application of attention mechanism – The transformer architecture Benefits of applying attention mechanism Real-world applications of attention mechanism Use of attention mechanism in machine translation Role in NLP tasks Applications of attention mechanism in computer vision The introduction of attention mechanism in deep learning Attention mechanism has emerged as a significant concept in deep learning. Although this mechanism is currently applied to diverse tasks such as image captioning, its original design was intended for application in neural machine translation using sequence-to-sequence models. How does seq2seq model work? 2/27

3. Consider a sequence modeling task with a variable-length input sequence; the objective is to predict a variable-length output sequence. A prevalent instance of this task is machine translation, such as translating “I love you” (English) to “te amo” (Spanish). One can observe that the length of the input sequence (3 words) doesn’t necessarily match the length of the output sequence (2 words). This discrepancy makes sequence prediction a complex task since it’s impossible to train a model to map each input token to a corresponding output token due to the lack of a 1:1 mapping. The RNN encoder–decoder neural network architecture, introduced by Cho et al., was devised to tackle this kind of sequence modeling. It employs a recurrent neural network to encode the variable-length input into a fixed-size vector. A second recurrent neural network then decodes this fixed-size vector into a variable-length output sequence. Building upon this approach, Sutskever et al. introduced a similar architecture that uses LSTM networks for encoding and decoding variable-length sequences. However, in this structure, only the final hidden state of the encoder is utilized to initialize the decoder instead of providing it as context for each step during decoding. Interestingly, the input sequence is reversed to simplify the optimization problem. Despite minor differences in the two architectures, the general approach remains the same: the variable-length input sequence is encoded into a fixed-size vector which encapsulates a summary of the entire input. This summary serves as the context for the decoder while generating the output sequence. Limitations of the context vector 3/27

4. While the fixed-size context vector paves the way for flexible learning of sequence-to- sequence tasks such as machine translation or text summarization, it’s crucial to consider the implications of this architectural choice. The vector’s fixed-size nature suggests that a finite amount of information can be stored as a summary of the input sequence. For instance, if we assume that our context vector can retain 100 bits of information and each input sequence token contributes equally to this information, then for an input sequence of 5 tokens, each token would contribute 20 bits of information to the context vector. However, for an input sequence of 10 tokens, each token would only offer 10 bits of information to the context vector. This illustrative example shows how the performance of sequence-to-sequence models may diminish for longer input sequences. As the size of the input increases, the information that can be stored per token in the input sequence progressively decreases. Introducing attention mechanism: The solution To tackle the issue of the ‘context bottleneck,’ Bahdanau et al. introduced a neural network architecture that facilitates the construction of a distinct context vector for every time step in the decoder. This is achieved based on varying weighted aggregations across all the hidden states in the encoder. In the previous illustrative example, we had only 100 bits of information to summarize the entire input sequence. Yet, with this new method, we can potentially give 100 bits of information about the input sequence at each phase of the decoding process 4/27

5. The attention mechanism produces the context vector by taking a weighted combination of all the hidden states of the encoder rather than just the final hidden state. The attention weights determine the relevance of the encoder’s hidden states given the current decoder’s hidden state. It provided the foundation for the transformative paper by Vaswani et al., “Attention is All You Need,” which brought a significant change in the field of deep learning with the concept of parallel processing of words. The essence of this approach is that, when predicting an output word, the model concentrates solely on the sections of the input that contain the most pertinent information, rather than considering the entire sequence. In other words, it ‘pays attention’ to certain input words. The attention mechanism acts as a link that binds the encoder and decoder, furnishing the decoder with data from each hidden state of the encoder. This selective focus on valuable parts of the input sequence allows the model to manage long input sentences efficiently. Recall our example of translating “I love you” into Spanish. We are essentially aligning the related tokens of “I love you” and “te amo.” Bahdanau et al. refer to this attention mechanism as an alignment model, allowing the decoder to search for the most relevant time steps across the input sequence, independent of the decoder model’s temporal position. When decoding the second token, we obtain a different weighted combination of contexts from the encoder’s hidden states, aligning “I love you” with “te amo.” As the decoder model continues to generate the output sequence, it will keep comparing the current decoder hidden state against all our encoder hidden states to produce a unique context vector. So, when the proposed model constructs a sentence, it identifies certain positions within the encoder’s hidden states that contain the most pertinent information. This selective focus on relevant areas is referred to as ‘attention.’ Launch your project with LeewayHertz! Elevate your AI experience with our Transformer-based solutions that use attention mechanisms to deliver unmatched accuracy, adaptability, and performance. Choose innovation, choose excellence! Learn More Types of attention mechanism Before we proceed to explore the intricate workings of the attention mechanism, it’s crucial to understand that there are different forms of attention mechanism. The differentiation among attention mechanisms is primarily based on their specific areas of application and the 5/27

6. portions of the input sequence to which the model concentrates or ‘pays attention.’ These forms are as follows: Bahdanau attention/ Additive attention The first type of attention mechanism, often termed as additive attention, emerged from a paper by Dzmitry Bahdanau. The study’s primary objective was to enhance the sequence-to- sequence model in machine translation by aligning the decoder with the pertinent input sentences and integrating attention. The complete procedure of applying attention, as illustrated in Bahdanau’s paper, is described below: 1. Initially, the encoder generates the hidden states for each element in the input sequence. 2. Subsequently, alignment scores are calculated between the previous decoder’s hidden state and each of the encoder’s hidden states. It’s important to note that the final encoder’s hidden state could be utilized as the initial hidden state in the decoder. 3. Next, the alignment scores for each encoder hidden state are amalgamated into a single vector, which is then subject to a softmax function. This essentially converts the raw alignment scores into probabilities that sum up to one, emphasizing certain hidden states over others. 4. The context vector is formed by multiplying the alignment scores with their corresponding hidden states of the encoder. This context vector essentially serves as a weighted sum of the encoder’s hidden states, with the weights determined by the alignment scores. 6/27

7. 5. The context vector is subsequently combined with the output from the previous decoding step. This combined information is fed into the decoder RNN at that time step, along with the previous decoder hidden state, to generate a new output. This process, encompassing steps 2 through 5, is repeated for each time step of the decoder until an end token is produced or the output exceeds a specified maximum length. Luong attention/ Global attention/ Multiplicative attention The second type of attention, frequently referred to as multiplicative attention, was suggested by Thang Luong in his paper. This approach builds on the attention mechanism proposed by Bahdanau. The primary differences between Luong attention and Bahdanau attention lie in the calculation of the alignment score and the integration point of the attention mechanism in the decoder. Luong’s paper proposes three distinct types of alignment scoring functions, in contrast to the single type in Bahdanau’s model. Furthermore, the general structure of the attention decoder varies in Luong attention, where the context vector is utilized only after the RNN has generated the output for that specific time step. Here is an outline of the Luong attention process: 1. Initially, the encoder produces hidden states for each element in the input sequence. 2. Subsequently, the previous decoder’s hidden state and decoder output are passed through the decoder RNN, generating a new hidden state for that time step. 7/27

8. 3. Next, alignment scores are calculated using the newly generated decoder and encoder hidden states. 4. Subsequently, the alignment scores for each encoder hidden state are compiled into a single vector. Then, a softmax function is applied, converting these raw alignment scores into probability values. 5. The context vector is then calculated by multiplying the encoder hidden states with their respective alignment scores. 6. Lastly, the context vector is concatenated with the decoder hidden state generated in step 2 and passed through a fully connected layer to produce a new output. This procedure, encompassing steps 2 through 6, is reiterated for each time step of the decoder until an end token is produced or the output surpasses a certain maximum length. The sequence of steps in Luong attention differs from that in Bahdanau attention. Self-attention The self-attention mechanism, also known as intra-attention, is a model that allows for the recognition and utilization of dependencies between all positions of the input sequence when forming the output sequence. This mechanism shines when dealing with sequences, as it 8/27

9. considers all parts of the input sequence irrespective of their positions, thus identifying relevant information that may not be contiguous. Unlike traditional attention mechanisms that generate context vectors based on the relationship between input (encoder) and output (decoder) sequences, the self-attention mechanism generates a context for each element of the sequence solely based on the sequence itself. This allows it to learn and understand the interdependencies and relationships between different parts of the sequence, all while minimizing the need for manual data entry or external assistance in predicting the output sequence. In the self-attention mechanism, each element in the input sequence has a corresponding weight. These weights determine the degree to which other elements in the sequence should influence the current element. As such, the self-attention mechanism can weigh the significance of each element in the sequence and adjust its impact on the generated output accordingly. For example, in the case of natural language processing, the self-attention mechanism allows each word in a sentence to influence every other word, helping the model understand the context and semantic relationship between words. This ability to understand and use the interdependencies in the sequence makes the self-attention mechanism an incredibly powerful tool for tasks such as text translation, text summarization, sentiment analysis, and more. Multi-head attention Multi-head attention is a specialized component of the transformer attention mechanism. The basic idea behind multi-head attention is the concept of parallel processing, where the attention mechanism is applied multiple times in parallel, creating distinct layers known as 9/27

10. ‘attention heads.’ Each of these attention heads independently processes the input sequence and its corresponding output sequence element. The unique aspect is that each head works with different learned linear transformations of the original sequences. This allows each attention head to focus on different features and aspects of the input sequence, enabling the model to capture a wider range of information and various nuances within the data. The attention scores, or weights, generated at each head are then combined to produce a final attention score. The process of combining involves concatenating the outputs of each head and passing this through a linear transformation. The resulting output maintains the same dimensionality as the input, but each output position now contains information from all input positions, as seen by multiple ‘heads’ or perspectives. This multi-head attention mechanism allows the model to focus on different parts of the input sequence simultaneously, providing a more comprehensive understanding of the data. This capacity is particularly useful for complex tasks where various types of information or contextual nuances are relevant, such as in natural language processing, image recognition, and more. How does attention mechanism work? While traditional attention mechanisms focus on interdependencies between input and output sequences, self-attention and multi-head attention bring unique perspectives by focusing on intra-sequence relationships and parallel processing. The following sections will delve deeper into these powerful attention variants, exploring how they redefine sequence understanding in AI models. Self-attention mechanism 10/27

11. Figure 1 Imagine the sentence, “Bark is incredibly adorable, and he is a dog.” This sentence consists of nine words or tokens. If we isolate the word ‘he,’ we see that ‘and’ and ‘is’ are the two adjacent words. However, these words don’t provide any meaningful context for ‘he.’ Instead, the words ‘Bark’ and ‘dog’ are more contextually relevant to ‘he.’ This example shows that in a sentence, context often takes precedence over proximity. When a computer processes this sentence, each word, or token, is given a word embedding, V. These initial embeddings, however, lack context. The goal is to apply weights or similarities to derive a final word embedding, Y, that is more context-rich than the original embedding, V. In the realm of embeddings, similar words tend to have similar embeddings, placing them closer together in the embedding space. For instance, ‘king’ will have a closer relationship and thus similar embedding to ‘queen’ and ‘royalty’ than ‘zebra.’ By the same token, ‘zebra’ will be closer to ‘horse’ and ‘stripes’ than ’emotion.’ This concept forms the basis for determining weight vectors, W. The idea is to multiply (via dot product) the word embeddings with each other to gain more context. Therefore, in our sentence, “Bark is incredibly adorable, and he is a dog,” instead of using the raw word embeddings, we calculate the dot product of embedding each word with all the others. Figure 2 should illustrate this better. 11/27

12. Figure 2 We first calculate the weights by taking the dot product of the initial embedding of the first word with the embeddings of all other words in the sentence. These weights (W11 to W19) are normalized so that they sum to 1. Then, these weights are multiplied by the initial embeddings of all the words in the sentence. Launch your project with LeewayHertz! Elevate your AI experience with our Transformer-based solutions that use attention mechanisms to deliver unmatched accuracy, adaptability, and performance. Choose innovation, choose excellence! Learn More W11 V1 + W12 V2 + …. W19 V9 = Y1 Here, W11 to W19 are all weights that incorporate the context of the first word, V1. So, when we multiply these weights with each word, we essentially reweigh all other words relative to the first word. This means the word ‘Bark’ is now more strongly associated with the words 12/27

13. ‘dog’ and ‘cute,’ as opposed to the immediate succeeding word. This procedure, therefore, provides context, as shown in Figure 3. Figure 3 This process is repeated for all words in the sentence; thus, each word is given context from every other word in the sentence. This technique of providing context to the words in a sentence is known as self-attention. One of the most interesting facets of self-attention is that it doesn’t rely on word order, word proximity, or sentence length, and it does not require training any weights. As depicted in Figure 3 of the self-attention mechanism, the initial word embeddings (V) appear thrice. They are first used to find the dot product between the initial word embedding and every other word in the sentence (including the word itself). This computation generates weights, which are then multiplied with the word embeddings for the third time to provide the final context-rich embeddings. In this scenario, we can replace these three occurrences of V with the terms Query, Keys, and Values. Assume we aim to align all the words in relation to the first word V1. We then designate V1 as the Query word. This Query word computes a dot product with every other word in the sentence (V1 to V9) – these are considered as the Keys. This interaction between the Query 13/27

14. and the Keys yields the weights. These weights are then multiplied with all the words once more (V1 to V9), serving as the Values. This process gives us the concepts of Query, Keys, and Values. For further clarity, you can refer to Figure 4. Figure 4 While the self-attention mechanism offers significant advantages in capturing the interdependencies within a sequence, there are some limitations or pitfalls associated with it: Computational cost: Self-attention requires the computation of pairwise interactions between all elements in a sequence, leading to quadratic computational complexity in terms of sequence length. This can be a substantial drawback when dealing with longer sequences, making it computationally expensive. 14/27

15. Lack of positional information: As self-attention computes the weightage based on the similarity between elements, it inherently lacks information about the absolute or relative positions of elements within the sequence. Although this issue can be mitigated using positional encodings or relative positional embeddings, it still presents an additional step to incorporate such positional information. Memory intensive: Self-attention mechanism demands a high amount of memory to store the attention maps, especially for longer sequences. This can pose a significant challenge when training large models or working with longer sequence lengths. Difficulty in modeling local dependencies: Despite its strength in capturing global dependencies, self-attention might face difficulties modeling local dependencies or exploiting locality information as effectively as convolutional or recurrent networks, especially when dealing with high-resolution inputs like images. Lack of interpretability: Although attention weights can sometimes provide some level of interpretability, it’s often difficult to fully understand and interpret the learned dependencies in self-attention. This is particularly true when dealing with multiple layers of attention or complex structures like multi-head attention. Risk of overfitting: With its ability to capture all pairwise dependencies, self-attention might overfit the training data, especially when dealing with smaller datasets. Despite these pitfalls, self-attention has proven to be a powerful mechanism in many applications, and ongoing research is continually finding solutions to these challenges. Multi-head attention mechanism Figure 5 15/27

16. The concept of multi-head attention comes into play to enhance the attention mechanism’s ability to focus on different aspects of the input simultaneously. Let’s revisit the sentence – “Bark is very cute and he is a dog.” When considering the word ‘dog’, grammatically, we understand that the words ‘Bark’, ‘cute’, and ‘he’ hold significance relative to ‘dog’. These words indicate that the dog’s name is Bark, he is male, and he is considered cute. A single attention mechanism may not correctly identify these three words as relevant to ‘dog’, suggesting that employing three separate attention mechanisms could be more effective at associating these words with ‘dog’. This approach reduces the burden on a single attention mechanism to identify all relevant words and increases the likelihood of detecting more pertinent words. To implement this, we add more linear layers to represent keys, queries, and values. These layers operate in parallel and carry independent weights. Thus, instead of generating a single output, each of the keys, queries, and values produces three outputs. These three sets of keys and queries result in three different sets of weights. Each of these weights then interacts with the three values through matrix multiplication, resulting in three distinct outputs. These three attention blocks are finally concatenated to yield a single final attention output. This process is demonstrated in Figure 5. However, the number three was arbitrarily selected for this example. In reality, the number of linear layers, or “heads”, could be any number. Each head provides a separate attention output, and these outputs are concatenated together, which is why it’s called “multi-head” attention. The simplified version of Figure 5, but with an arbitrary number of heads, is shown in Figure 6. 16/27

17. Figure 6 Application of attention mechanism – The transformer architecture The groundbreaking research document ‘Attention Is All You Need’ presents a unique structure known as the “transformer.” As the name suggests, this model employs the previously mentioned attention mechanism. Comparable to LSTM, the transformer is designed to convert one sequence into another utilizing two components (the encoder and the decoder), yet it distinctively departs from existing sequence-to-sequence models by not incorporating any recurrent networks (such as GRU, LSTM, etc.). Recurrent networks have been considered among the most effective methods for handling temporal dependencies in sequences. However, the research group behind this paper has demonstrated that a framework solely employing attention mechanisms and completely excluding any form of Recurrent Neural Networks (RNNs) can outperform existing models in tasks such as translation, among others. Thus, we ask, what exactly constitutes a transformer? 17/27

18. Figure 7 The encoder module is positioned on the left, and the decoder on the right. Both the encoder and decoder are made up of modules that can be repeatedly stacked, as denoted by Nx in the diagram. The main components of these modules are multi-head attention and feed- forward layers. Because we cannot work with strings directly, both the input and output (the target sentences) are initially mapped to n-dimensional space. 18/27

19. A subtly crucial aspect of this model is the positional encoding attributed to each word. In the absence of recurrent networks capable of remembering the order in which sequences are input into the model, assigning a relative position to every part/word in our sequence is necessary since the sequence’s significance depends on the order of its components. These positions are appended to the n-dimensional vector representing each word’s embedded form. Now, let’s delve a little deeper into these multi-head attention components of the model: Figure 8 Let’s commence with the leftmost illustration of the attention mechanism. It’s quite straightforward and can be expressed using the following formula: Launch your project with LeewayHertz! Elevate your AI experience with our Transformer-based solutions that use attention mechanisms to deliver unmatched accuracy, adaptability, and performance. Choose innovation, choose excellence! 19/27

20. Learn More Q represents a matrix holding the query (a vectorized representation of a single word in the sequence), K contains all the keys (vectorized representations of every word in the sequence), and V signifies the values, which are yet again vectorized representations of all words in the sequence. For the multi-head attention modules in both the encoder and decoder, V comprises the same word sequence as Q. Nevertheless, in the attention module that considers both the encoder and decoder sequences, V is different from the sequence represented by Q. To simplify this, it can be stated that the values in V are multiplied and summed with certain attention weights, where weights are determined by how each word of the sequence (represented by Q) is influenced by all other words in the sequence (represented by K). Furthermore, a SoftMax function is employed on the weights a to ensure a distribution between 0 and 1. These weights are then applied to all words in the sequence present in V (identical vectors as Q for encoder and decoder, but different for the module that includes inputs from both encoder and decoder). The right-side diagram illustrates how this attention mechanism can be distributed into multiple parallel mechanisms for simultaneous use. The attention mechanism is repeated multiple times with linear projections of Q, K, and V. This capability enables the system to learn from varied Q, K, and V representations, which benefits the model. These linear representations are performed by multiplying Q, K, and V with weight matrices W, learned during training. The matrices Q, K, and V vary for each position of the attention modules within the architecture based on whether they are situated in the encoder, decoder, or the area bridging the encoder and decoder. This variation arises because we aim to focus either on the entire input sequence of the encoder or a portion of the input sequence from the decoder. The multi-head attention module linking the encoder and decoder ensures that the encoder’s input sequence is considered alongside the decoder’s input sequence up to a certain position. Following the multi-attention heads in both the encoder and decoder, a pointwise feed- forward layer exists. This compact feed-forward network maintains identical parameters for each position, which can be interpreted as a separate yet identical linear transformation of each element from the provided sequence. In a nutshell – here is a step-by-step walkthrough of how the Transformer architecture works: 20/27

21. 1. Attention mechanism: The attention mechanism enables the transformer to “attend” or “focus” on all previous tokens that have been generated, unlike RNNs and their variants that have a limited window to reference from. Maintaining a virtually unlimited memory window allows the transformer to understand and use the entire context while generating text. 2. Input embeddings: In the initial step, the input is fed into a word embedding layer. This layer acts as a lookup table to fetch a learned vector representation of each word. The word embedding layer converts words into continuous value vectors to make them understandable by neural networks. 3. Positional encoding: The next step involves injecting positional information into the embeddings. Since transformer encoders don’t have recurrence like RNNs, it’s important to add information about the position of each word in the sequence. 4. Encoder and decoder: The transformer architecture is essentially an attention-based encoder-decoder type architecture. Both the encoder and decoder stacks have their corresponding embedding layers. An additional output layer is used to generate the final output. Each encoder and decoder has its own set of weights. The encoder maps an input sequence into a continuous abstract representation that contains all the learned information of that input. The decoder then uses that representation to generate an output, one step at a time. It also considers the previous output while generating the next output. 5. Self-attention and feed-forward layers: The encoder comprises the crucial self- attention layer that calculates the relationship between different words in the sequence and a feed-forward layer. The decoder contains a self-attention layer, a feed-forward layer, and an additional encoder-decoder attention layer. The strength of the Transformer model lies in its use of attention, which allows the model to focus on words closely related to the word it’s processing. The model can better understand the context and generate more relevant outputs by attending to the important and related words in a sentence. Benefits of employing attention mechanism Applying the attention mechanism in machine learning models, such as in the transformer architecture, carries many benefits that enhance their overall performance. Here are some key advantages: Capturing long-term dependencies: Some input elements can depend on distant elements in sequence-to-sequence tasks. Traditional recurrent neural networks can struggle with such long-term dependencies due to the “vanishing gradient” problem. Attention mechanisms can alleviate this issue by allowing the model to focus on relevant parts of the input sequence, regardless of their position, enabling it to capture these long-range dependencies better. 21/27

22. Interpretability: The attention mechanism allows us to visualize the weights the model assigns to different inputs when making predictions. This ability to interpret the model’s decision-making process provides insight into what the model considers important or relevant, which can be critical in certain domains, such as healthcare or finance. Efficiency: Traditional sequence-to-sequence models often require significant computational resources because they process inputs sequentially. On the other hand, the attention mechanism allows the model to parallelize computation as it can process all inputs simultaneously, leading to efficiency gains. Performance improvement: Empirically, models with attention mechanisms often outperform those without. The ability to weigh the importance of different inputs allows the model to make more informed decisions, leading to more accurate predictions. This has been particularly evident in machine translation and speech recognition tasks. Handling variable-length input and output: Attention mechanisms excel in tasks where the input and output sequences have different lengths. This property is particularly useful in machine translation, where the length of the source and target sentences can vary greatly. Context-awareness: By weighing the importance of different parts of the input sequence when generating each part of the output sequence, attention mechanisms allow the model to consider the broader context of the input data, leading to more nuanced and accurate predictions. Reduced burden on encoding: In traditional encoder-decoder structures, the encoder must represent the entirety of the input in a fixed-length vector, which is then used by the decoder to generate the output. This can be a significant limitation when dealing with longer sequences. The attention mechanism reduces the burden on the encoder by allowing the decoder to ‘look back’ at the input sequence, thereby spreading the representational load. The attention mechanism significantly enhances the model’s capability to understand, process, and predict from sequence data, especially when dealing with long, complex sequences. Its ability to interpret, increase performance, and efficiently utilize computational resources has made it a cornerstone of many modern machine learning architectures. Real-world applications of attention mechanism Use of attention mechanism in machine translation When generating the output sequence, the decoder considers the encoded input vector and the previously generated words and computes an attention score for each word in the input sequence. These attention scores determine how much ‘attention’ should be paid to each input word while generating the current output word. 22/27

23. In practical terms, this means that while translating a sentence, if the model is currently generating a word that directly corresponds to a specific word in the source sentence, the attention mechanism allows the model to focus on that source word, regardless of its position in the sentence resulting in the following benefits: Improved translation quality: By allowing the model to refer back to the input sequence, the attention mechanism helps to produce more accurate and fluent translations. Handling of long sentences: The attention mechanism mitigates the issue of information loss in long sentences, which was a notable problem in traditional seq2seq models. Alignment information: The attention scores can provide insight into the alignment between the source and target sentences, showing which source words were considered important when generating each target word. Role in NLP tasks Attention mechanisms have been a game-changer in many NLP tasks such as text summarization, question answering, sentiment analysis, etc. Here is how: Text summarization: Text summarization is the process of generating a concise and meaningful summary of a longer text document. The attention mechanism plays a crucial role in identifying the most important parts of the input text to include in the summary. In a seq2seq model applied to text summarization, the attention mechanism allows the model to look back at the original text (instead of relying on the encoded state) when generating the summary. When generating each word of the summary, the attention mechanism computes attention scores for each word in the source text. These scores essentially indicate the relevance or contribution of each source word to the word being generated in the summary. This allows the model to focus on the most relevant parts of the original text at each step of summary generation, thereby creating summaries that capture the most important information from the original text. Question answering: Question Answering (QA) is an NLP task requiring the model to answer questions about a given text. Attention mechanisms have significantly improved the performance of QA models. Attention allows the model to focus on the relevant parts of the text when generating an answer. When presented with a question and a context (for example, a paragraph containing the answer), the model computes attention scores for each word in the context. These scores highlight the words in the context most relevant to the question. The model then uses these highlighted parts to generate the answer, which increases the chances of the answer being correct. 23/27

24. Other NLP tasks: In many other NLP tasks, the attention mechanism helps the model to focus on the important parts of the input: Named Entity Recognition (NER): In NER, the model identifies proper nouns in the text (like person names, organization names, and locations). Attention can help the model focus on the context around a word, which aids in classifying the word correctly. Sentiment analysis: For tasks like sentiment analysis, the sentiment of a sentence is often determined by a few keywords or phrases. The attention mechanism can help the model focus on these key parts when determining the sentiment. The attention mechanism allows NLP models to dynamically focus on the most relevant parts of the input data at each step of processing, leading to improved performance on many complex tasks. By assigning different weights to different parts of the input, attention mechanisms can capture dependencies in the data that other models may miss. Launch your project with LeewayHertz! Elevate your AI experience with our Transformer-based solutions that use attention mechanisms to deliver unmatched accuracy, adaptability, and performance. Choose innovation, choose excellence! Learn More Applications of attention mechanism in computer vision The attention mechanism, which has seen tremendous success in natural language processing, has also been applied in computer vision with promising results. In computer vision, it can be used to guide a model’s focus toward certain regions in the image when performing tasks such as image classification, object detection, or semantic segmentation. How the attention mechanism works in computer vision: Spatial attention: In spatial attention, the model learns to focus on specific regions of the image. An attention score is computed for each region in the image, usually based on the features extracted from that region. These scores determine how much ‘attention’ each region should receive. The features from each region are then weighted according to their attention scores before being passed on to the next layer of the model. This allows the model to focus more on the regions more relevant to the task at hand. 24/27

25. Channel attention: Channel attention focuses on selecting more informative feature channels. For each channel of the feature map, an attention score is computed, which measures the importance of that channel. The channels are then weighted according to their attention scores, allowing the model to focus more on the channels most relevant to the task at hand. Self-attention in computer vision: Self-attention, also known as non-local operation, computes the response at a position in an image as a weighted sum of the features at all positions in the image. This allows the model to consider the global context rather than just the local context provided by traditional convolutional operations. Applications of attention mechanism in computer vision: Object detection and segmentation: In tasks like object detection and segmentation, spatial attention can help the model focus on the regions containing the objects of interest while ignoring the background or irrelevant objects. Image captioning: In image captioning, where the task is to generate a textual description of an image, attention mechanism can help the model focus on different regions of the image at each step of the caption generation. This allows the model to generate more accurate and detailed captions. Visual question answering: In visual question answering, where the model is given an image and a question about the image (for example, “What color is the cat in the image?”), attention mechanism can help the model focus on the relevant parts of the image when generating the answer. Transformers in vision: Inspired by the success of transformers in NLP, researchers have also applied transformers to computer vision tasks. For example, the Vision Transformer (ViT) treats an image as a sequence of patches and applies self-attention and position embeddings, much like how a transformer works with a sequence of tokens in NLP. Attention mechanism allows computer vision models to dynamically allocate their computational resources, focusing more on the important parts of the image or feature map and less on the less relevant parts. This can lead to improved performance on a variety of complex tasks. Endnote Conventional encoder-decoder sequence models often grapple with an information bottleneck. This problem arises when transferring information from the encoder to the decoder phases. The attention mechanism serves as a strategic solution to this predicament, allowing the decoder to navigate through the entirety of the input sequence while generating each component of the output sequence. 25/27

26. This is achieved by integrating a compact attention model designed to calculate a relevance score correlating each encoder and decoder hidden state. Leveraging these relevance scores, a distinctive weighted amalgamation of encoder hidden states can be constructed as context for each stage of the decoding process. The advent of the attention mechanism has indisputably transformed the landscape of NLP model creation. Its application has become a standard constituent of cutting-edge NLP models. This can be attributed to its ability to “remember” all words in the input and concentrate on specific words while composing a response, thereby greatly enhancing the model’s effectiveness and accuracy. Ready to harness the power of AI for your business? Contact LeewayHertz’s AI experts today and for smart AI solutions tailored to your needs! Listen to the article 23:10 00:00 Author’s Bio Akash Takyar CEO LeewayHertz 26/27

27. Akash Takyar is the founder and CEO at LeewayHertz. The experience of building over 100+ platforms for startups and enterprises allows Akash to rapidly architect and design solutions that are scalable and beautiful. Akash's ability to build enterprise-grade technology solutions has attracted over 30 Fortune 500 companies, including Siemens, 3M, P&G and Hershey’s. Akash is an early adopter of new technology, a passionate technology enthusiast, and an investor in AI and IoT startups. Write to Akash Start a conversation by filling the form Once you let us know your requirement, our technical expert will schedule a call and discuss your idea in detail post sign of an NDA. All information will be kept confidential. Insights Show all Insights 27/27