Chapter 10: Machine Translation
10.1 Sequence to Sequence Models
In the vast world of Natural Language Processing, Machine Translation holds a pivotal place. It's a subfield that focuses on translating text from one language to another, an incredibly important task in today's globalized world. The need for effective machine translation has only grown in recent years, as more and more people around the world seek to communicate with each other, whether for business or personal reasons. Machine Translation has a wide array of uses, from aiding international diplomacy to assisting in the translation of large volumes of text for academic or business purposes.
The advent of deep learning has brought a revolutionary change to the field of Machine Translation. Models that leverage this technique are able to learn complex language representations and generate translations that are often remarkably close to human ones, making it possible to bridge the gap between languages and cultures. This not only facilitates communication, but also has the potential to foster greater understanding and cooperation between people from different backgrounds.
In this chapter, we will delve into the inner workings of Machine Translation, focusing on some of the most effective techniques and models used today. We'll start with a deep dive into sequence-to-sequence (Seq2Seq) models, which form the backbone of many modern machine translation systems. We'll then explore how attention mechanisms have further improved the performance of these models, allowing them to selectively focus on different parts of the input sequence to generate more accurate translations.
Finally, we'll discuss some of the challenges that still remain in the field of Machine Translation, and the exciting opportunities that lie ahead as researchers continue to push the boundaries of what is possible.
Sequence-to-sequence (Seq2Seq) models, as the name suggests, transform an input sequence into an output sequence. These models are particularly effective in tasks like Machine Translation, where the length of the input sequence (source language text) can differ from that of the output sequence (translated text).
The Seq2Seq model is composed of two main components: an encoder and a decoder.
10.1.1 Encoder
The encoder plays a crucial role in natural language processing (NLP) tasks. It processes the input sequence and compresses the information into a context vector, also known as the hidden state. This context vector is a representation of the entire input sequence and contains the inherent meaning of the text.
There are several types of encoders used in NLP tasks. The most commonly used type is a Recurrent Neural Network (RNN). An RNN is a type of neural network that is capable of processing sequential data. This makes it an ideal choice for NLP tasks as it can handle variable-length sequences of text. In addition to RNNs, other types of encoders include Convolutional Neural Networks (CNNs) and Transformer-based models.
CNNs are commonly used for tasks such as text classification where the focus is on identifying the most relevant feature of the input text. Transformer-based models, on the other hand, have become popular in recent years due to their ability to capture long-range dependencies in text. They are particularly useful for tasks such as language translation where the input and output sequences can be of different lengths.
Despite the differences in their design, all encoders play a critical role in NLP tasks by extracting meaningful information from raw text. This information can then be used for a wide range of applications such as sentiment analysis, chatbots, and machine translation.
Example:
Here is an example code snippet showing how to implement an encoder using LSTM, a type of RNN, in PyTorch:
import torch
import torch.nn as nn
class Encoder(nn.Module):
def __init__(self, input_dim, emb_dim, hid_dim, n_layers, dropout):
super().__init__()
self.hid_dim = hid_dim
self.n_layers = n_layers
self.embedding = nn.Embedding(input_dim, emb_dim)
self.rnn = nn.LSTM(emb_dim, hid_dim, n_layers, dropout=dropout)
self.dropout = nn.Dropout(dropout)
def forward(self, src):
embedded = self.dropout(self.embedding(src))
outputs, (hidden, cell) = self.rnn(embedded)
return hidden, cell10.1.2 Decoder
The decoder is a crucial component in neural machine translation models. It takes the context vector, which is produced by the encoder, and uses it to generate the output sequence. Generally, the decoder is also an RNN. The decoder accomplishes this task step-by-step, generating one token at each step. It utilizes the context vector and all of the tokens that have been generated up to that point as input.
It is important to note that the decoder is responsible for generating the output sequence, which is in turn responsible for translating the source language into the target language. Without the decoder, the neural machine translation model would be incomplete and unable to produce a translation. Therefore, the decoder plays a significant role in the success of the entire neural machine translation process.
Example:
Here's an example of a decoder implementation:
class Decoder(nn.Module):
def __init__(self, output_dim, emb_dim, hid_dim, n_layers, dropout):
super().__init__()
self.output_dim = output_dim
self.hid_dim = hid_dim
self.n_layers = n_layers
self.embedding = nn.Embedding(output_dim, emb_dim)
self.rnn = nn.LSTM(emb_dim, hid_dim, n_layers, dropout=dropout)
self.fc_out = nn.Linear(hid_dim, output_dim)
self.dropout = nn.Dropout(dropout)
def forward(self, input, hidden, cell):
input = input.unsqueeze(0)
embedded = self.dropout(self.embedding(input))
output, (hidden, cell) = self.rnn(embedded, (hidden, cell))
prediction = self.fc_out(output.squeeze(0))
return prediction, hidden, cell10.1.3 Seq2Seq Model
The Seq2Seq model is a powerful tool in natural language processing. It consists of two main components, namely the encoder and decoder, which work together to produce the desired output.
The encoder takes in the input sequence and produces a context vector, which captures the essential information needed for the decoding stage. In the decoding stage, the decoder uses this context vector to generate the output sequence, one step at a time.
It is important to note that the Seq2Seq model has many practical applications, including machine translation, text summarization, and speech recognition, among others. In fact, it is widely used in the industry due to its effectiveness and versatility.
Example:
Here's a simple implementation of a Seq2Seq model in PyTorch:
class Seq2Seq(nn.Module):
def __init__(self, encoder, decoder, device):
super().__init__()
self.encoder = encoder
self.decoder = decoder
self.device = device
def forward(self, src, trg, teacher_forcing_ratio=0.5):
batch_size = trg.shape[1]
trg_len = trg.shape[0]
trg_vocab_size = self.decoder.output_dim
outputs = torch.zeros(trg_len, batch_size, trg_vocab_size).to(self.device)
hidden, cell = self.encoder(src)
input = trg[0,:]
for t in range(1, trg_len):
output, hidden, cell = self.decoder(input, hidden, cell)
outputs[t] = output
teacher_force = random.random() < teacher_forcing_ratio
top1 = output.argmax(1)
input = trg[t] if teacher_force else top1
return outputsThis code snippet defines a Seq2Seq model that uses the previously defined encoder and decoder. The model uses a technique called teacher forcing during training. Teacher forcing is a strategy for training sequence to sequence models that uses the true output sequence as input to the decoder rather than the output generated by the decoder in the previous time step.
10.1 Sequence to Sequence Models
In the vast world of Natural Language Processing, Machine Translation holds a pivotal place. It's a subfield that focuses on translating text from one language to another, an incredibly important task in today's globalized world. The need for effective machine translation has only grown in recent years, as more and more people around the world seek to communicate with each other, whether for business or personal reasons. Machine Translation has a wide array of uses, from aiding international diplomacy to assisting in the translation of large volumes of text for academic or business purposes.
The advent of deep learning has brought a revolutionary change to the field of Machine Translation. Models that leverage this technique are able to learn complex language representations and generate translations that are often remarkably close to human ones, making it possible to bridge the gap between languages and cultures. This not only facilitates communication, but also has the potential to foster greater understanding and cooperation between people from different backgrounds.
In this chapter, we will delve into the inner workings of Machine Translation, focusing on some of the most effective techniques and models used today. We'll start with a deep dive into sequence-to-sequence (Seq2Seq) models, which form the backbone of many modern machine translation systems. We'll then explore how attention mechanisms have further improved the performance of these models, allowing them to selectively focus on different parts of the input sequence to generate more accurate translations.
Finally, we'll discuss some of the challenges that still remain in the field of Machine Translation, and the exciting opportunities that lie ahead as researchers continue to push the boundaries of what is possible.
Sequence-to-sequence (Seq2Seq) models, as the name suggests, transform an input sequence into an output sequence. These models are particularly effective in tasks like Machine Translation, where the length of the input sequence (source language text) can differ from that of the output sequence (translated text).
The Seq2Seq model is composed of two main components: an encoder and a decoder.
10.1.1 Encoder
The encoder plays a crucial role in natural language processing (NLP) tasks. It processes the input sequence and compresses the information into a context vector, also known as the hidden state. This context vector is a representation of the entire input sequence and contains the inherent meaning of the text.
There are several types of encoders used in NLP tasks. The most commonly used type is a Recurrent Neural Network (RNN). An RNN is a type of neural network that is capable of processing sequential data. This makes it an ideal choice for NLP tasks as it can handle variable-length sequences of text. In addition to RNNs, other types of encoders include Convolutional Neural Networks (CNNs) and Transformer-based models.
CNNs are commonly used for tasks such as text classification where the focus is on identifying the most relevant feature of the input text. Transformer-based models, on the other hand, have become popular in recent years due to their ability to capture long-range dependencies in text. They are particularly useful for tasks such as language translation where the input and output sequences can be of different lengths.
Despite the differences in their design, all encoders play a critical role in NLP tasks by extracting meaningful information from raw text. This information can then be used for a wide range of applications such as sentiment analysis, chatbots, and machine translation.
Example:
Here is an example code snippet showing how to implement an encoder using LSTM, a type of RNN, in PyTorch:
import torch
import torch.nn as nn
class Encoder(nn.Module):
def __init__(self, input_dim, emb_dim, hid_dim, n_layers, dropout):
super().__init__()
self.hid_dim = hid_dim
self.n_layers = n_layers
self.embedding = nn.Embedding(input_dim, emb_dim)
self.rnn = nn.LSTM(emb_dim, hid_dim, n_layers, dropout=dropout)
self.dropout = nn.Dropout(dropout)
def forward(self, src):
embedded = self.dropout(self.embedding(src))
outputs, (hidden, cell) = self.rnn(embedded)
return hidden, cell10.1.2 Decoder
The decoder is a crucial component in neural machine translation models. It takes the context vector, which is produced by the encoder, and uses it to generate the output sequence. Generally, the decoder is also an RNN. The decoder accomplishes this task step-by-step, generating one token at each step. It utilizes the context vector and all of the tokens that have been generated up to that point as input.
It is important to note that the decoder is responsible for generating the output sequence, which is in turn responsible for translating the source language into the target language. Without the decoder, the neural machine translation model would be incomplete and unable to produce a translation. Therefore, the decoder plays a significant role in the success of the entire neural machine translation process.
Example:
Here's an example of a decoder implementation:
class Decoder(nn.Module):
def __init__(self, output_dim, emb_dim, hid_dim, n_layers, dropout):
super().__init__()
self.output_dim = output_dim
self.hid_dim = hid_dim
self.n_layers = n_layers
self.embedding = nn.Embedding(output_dim, emb_dim)
self.rnn = nn.LSTM(emb_dim, hid_dim, n_layers, dropout=dropout)
self.fc_out = nn.Linear(hid_dim, output_dim)
self.dropout = nn.Dropout(dropout)
def forward(self, input, hidden, cell):
input = input.unsqueeze(0)
embedded = self.dropout(self.embedding(input))
output, (hidden, cell) = self.rnn(embedded, (hidden, cell))
prediction = self.fc_out(output.squeeze(0))
return prediction, hidden, cell10.1.3 Seq2Seq Model
The Seq2Seq model is a powerful tool in natural language processing. It consists of two main components, namely the encoder and decoder, which work together to produce the desired output.
The encoder takes in the input sequence and produces a context vector, which captures the essential information needed for the decoding stage. In the decoding stage, the decoder uses this context vector to generate the output sequence, one step at a time.
It is important to note that the Seq2Seq model has many practical applications, including machine translation, text summarization, and speech recognition, among others. In fact, it is widely used in the industry due to its effectiveness and versatility.
Example:
Here's a simple implementation of a Seq2Seq model in PyTorch:
class Seq2Seq(nn.Module):
def __init__(self, encoder, decoder, device):
super().__init__()
self.encoder = encoder
self.decoder = decoder
self.device = device
def forward(self, src, trg, teacher_forcing_ratio=0.5):
batch_size = trg.shape[1]
trg_len = trg.shape[0]
trg_vocab_size = self.decoder.output_dim
outputs = torch.zeros(trg_len, batch_size, trg_vocab_size).to(self.device)
hidden, cell = self.encoder(src)
input = trg[0,:]
for t in range(1, trg_len):
output, hidden, cell = self.decoder(input, hidden, cell)
outputs[t] = output
teacher_force = random.random() < teacher_forcing_ratio
top1 = output.argmax(1)
input = trg[t] if teacher_force else top1
return outputsThis code snippet defines a Seq2Seq model that uses the previously defined encoder and decoder. The model uses a technique called teacher forcing during training. Teacher forcing is a strategy for training sequence to sequence models that uses the true output sequence as input to the decoder rather than the output generated by the decoder in the previous time step.
10.1 Sequence to Sequence Models
In the vast world of Natural Language Processing, Machine Translation holds a pivotal place. It's a subfield that focuses on translating text from one language to another, an incredibly important task in today's globalized world. The need for effective machine translation has only grown in recent years, as more and more people around the world seek to communicate with each other, whether for business or personal reasons. Machine Translation has a wide array of uses, from aiding international diplomacy to assisting in the translation of large volumes of text for academic or business purposes.
The advent of deep learning has brought a revolutionary change to the field of Machine Translation. Models that leverage this technique are able to learn complex language representations and generate translations that are often remarkably close to human ones, making it possible to bridge the gap between languages and cultures. This not only facilitates communication, but also has the potential to foster greater understanding and cooperation between people from different backgrounds.
In this chapter, we will delve into the inner workings of Machine Translation, focusing on some of the most effective techniques and models used today. We'll start with a deep dive into sequence-to-sequence (Seq2Seq) models, which form the backbone of many modern machine translation systems. We'll then explore how attention mechanisms have further improved the performance of these models, allowing them to selectively focus on different parts of the input sequence to generate more accurate translations.
Finally, we'll discuss some of the challenges that still remain in the field of Machine Translation, and the exciting opportunities that lie ahead as researchers continue to push the boundaries of what is possible.
Sequence-to-sequence (Seq2Seq) models, as the name suggests, transform an input sequence into an output sequence. These models are particularly effective in tasks like Machine Translation, where the length of the input sequence (source language text) can differ from that of the output sequence (translated text).
The Seq2Seq model is composed of two main components: an encoder and a decoder.
10.1.1 Encoder
The encoder plays a crucial role in natural language processing (NLP) tasks. It processes the input sequence and compresses the information into a context vector, also known as the hidden state. This context vector is a representation of the entire input sequence and contains the inherent meaning of the text.
There are several types of encoders used in NLP tasks. The most commonly used type is a Recurrent Neural Network (RNN). An RNN is a type of neural network that is capable of processing sequential data. This makes it an ideal choice for NLP tasks as it can handle variable-length sequences of text. In addition to RNNs, other types of encoders include Convolutional Neural Networks (CNNs) and Transformer-based models.
CNNs are commonly used for tasks such as text classification where the focus is on identifying the most relevant feature of the input text. Transformer-based models, on the other hand, have become popular in recent years due to their ability to capture long-range dependencies in text. They are particularly useful for tasks such as language translation where the input and output sequences can be of different lengths.
Despite the differences in their design, all encoders play a critical role in NLP tasks by extracting meaningful information from raw text. This information can then be used for a wide range of applications such as sentiment analysis, chatbots, and machine translation.
Example:
Here is an example code snippet showing how to implement an encoder using LSTM, a type of RNN, in PyTorch:
import torch
import torch.nn as nn
class Encoder(nn.Module):
def __init__(self, input_dim, emb_dim, hid_dim, n_layers, dropout):
super().__init__()
self.hid_dim = hid_dim
self.n_layers = n_layers
self.embedding = nn.Embedding(input_dim, emb_dim)
self.rnn = nn.LSTM(emb_dim, hid_dim, n_layers, dropout=dropout)
self.dropout = nn.Dropout(dropout)
def forward(self, src):
embedded = self.dropout(self.embedding(src))
outputs, (hidden, cell) = self.rnn(embedded)
return hidden, cell10.1.2 Decoder
The decoder is a crucial component in neural machine translation models. It takes the context vector, which is produced by the encoder, and uses it to generate the output sequence. Generally, the decoder is also an RNN. The decoder accomplishes this task step-by-step, generating one token at each step. It utilizes the context vector and all of the tokens that have been generated up to that point as input.
It is important to note that the decoder is responsible for generating the output sequence, which is in turn responsible for translating the source language into the target language. Without the decoder, the neural machine translation model would be incomplete and unable to produce a translation. Therefore, the decoder plays a significant role in the success of the entire neural machine translation process.
Example:
Here's an example of a decoder implementation:
class Decoder(nn.Module):
def __init__(self, output_dim, emb_dim, hid_dim, n_layers, dropout):
super().__init__()
self.output_dim = output_dim
self.hid_dim = hid_dim
self.n_layers = n_layers
self.embedding = nn.Embedding(output_dim, emb_dim)
self.rnn = nn.LSTM(emb_dim, hid_dim, n_layers, dropout=dropout)
self.fc_out = nn.Linear(hid_dim, output_dim)
self.dropout = nn.Dropout(dropout)
def forward(self, input, hidden, cell):
input = input.unsqueeze(0)
embedded = self.dropout(self.embedding(input))
output, (hidden, cell) = self.rnn(embedded, (hidden, cell))
prediction = self.fc_out(output.squeeze(0))
return prediction, hidden, cell10.1.3 Seq2Seq Model
The Seq2Seq model is a powerful tool in natural language processing. It consists of two main components, namely the encoder and decoder, which work together to produce the desired output.
The encoder takes in the input sequence and produces a context vector, which captures the essential information needed for the decoding stage. In the decoding stage, the decoder uses this context vector to generate the output sequence, one step at a time.
It is important to note that the Seq2Seq model has many practical applications, including machine translation, text summarization, and speech recognition, among others. In fact, it is widely used in the industry due to its effectiveness and versatility.
Example:
Here's a simple implementation of a Seq2Seq model in PyTorch:
class Seq2Seq(nn.Module):
def __init__(self, encoder, decoder, device):
super().__init__()
self.encoder = encoder
self.decoder = decoder
self.device = device
def forward(self, src, trg, teacher_forcing_ratio=0.5):
batch_size = trg.shape[1]
trg_len = trg.shape[0]
trg_vocab_size = self.decoder.output_dim
outputs = torch.zeros(trg_len, batch_size, trg_vocab_size).to(self.device)
hidden, cell = self.encoder(src)
input = trg[0,:]
for t in range(1, trg_len):
output, hidden, cell = self.decoder(input, hidden, cell)
outputs[t] = output
teacher_force = random.random() < teacher_forcing_ratio
top1 = output.argmax(1)
input = trg[t] if teacher_force else top1
return outputsThis code snippet defines a Seq2Seq model that uses the previously defined encoder and decoder. The model uses a technique called teacher forcing during training. Teacher forcing is a strategy for training sequence to sequence models that uses the true output sequence as input to the decoder rather than the output generated by the decoder in the previous time step.
10.1 Sequence to Sequence Models
In the vast world of Natural Language Processing, Machine Translation holds a pivotal place. It's a subfield that focuses on translating text from one language to another, an incredibly important task in today's globalized world. The need for effective machine translation has only grown in recent years, as more and more people around the world seek to communicate with each other, whether for business or personal reasons. Machine Translation has a wide array of uses, from aiding international diplomacy to assisting in the translation of large volumes of text for academic or business purposes.
The advent of deep learning has brought a revolutionary change to the field of Machine Translation. Models that leverage this technique are able to learn complex language representations and generate translations that are often remarkably close to human ones, making it possible to bridge the gap between languages and cultures. This not only facilitates communication, but also has the potential to foster greater understanding and cooperation between people from different backgrounds.
In this chapter, we will delve into the inner workings of Machine Translation, focusing on some of the most effective techniques and models used today. We'll start with a deep dive into sequence-to-sequence (Seq2Seq) models, which form the backbone of many modern machine translation systems. We'll then explore how attention mechanisms have further improved the performance of these models, allowing them to selectively focus on different parts of the input sequence to generate more accurate translations.
Finally, we'll discuss some of the challenges that still remain in the field of Machine Translation, and the exciting opportunities that lie ahead as researchers continue to push the boundaries of what is possible.
Sequence-to-sequence (Seq2Seq) models, as the name suggests, transform an input sequence into an output sequence. These models are particularly effective in tasks like Machine Translation, where the length of the input sequence (source language text) can differ from that of the output sequence (translated text).
The Seq2Seq model is composed of two main components: an encoder and a decoder.
10.1.1 Encoder
The encoder plays a crucial role in natural language processing (NLP) tasks. It processes the input sequence and compresses the information into a context vector, also known as the hidden state. This context vector is a representation of the entire input sequence and contains the inherent meaning of the text.
There are several types of encoders used in NLP tasks. The most commonly used type is a Recurrent Neural Network (RNN). An RNN is a type of neural network that is capable of processing sequential data. This makes it an ideal choice for NLP tasks as it can handle variable-length sequences of text. In addition to RNNs, other types of encoders include Convolutional Neural Networks (CNNs) and Transformer-based models.
CNNs are commonly used for tasks such as text classification where the focus is on identifying the most relevant feature of the input text. Transformer-based models, on the other hand, have become popular in recent years due to their ability to capture long-range dependencies in text. They are particularly useful for tasks such as language translation where the input and output sequences can be of different lengths.
Despite the differences in their design, all encoders play a critical role in NLP tasks by extracting meaningful information from raw text. This information can then be used for a wide range of applications such as sentiment analysis, chatbots, and machine translation.
Example:
Here is an example code snippet showing how to implement an encoder using LSTM, a type of RNN, in PyTorch:
import torch
import torch.nn as nn
class Encoder(nn.Module):
def __init__(self, input_dim, emb_dim, hid_dim, n_layers, dropout):
super().__init__()
self.hid_dim = hid_dim
self.n_layers = n_layers
self.embedding = nn.Embedding(input_dim, emb_dim)
self.rnn = nn.LSTM(emb_dim, hid_dim, n_layers, dropout=dropout)
self.dropout = nn.Dropout(dropout)
def forward(self, src):
embedded = self.dropout(self.embedding(src))
outputs, (hidden, cell) = self.rnn(embedded)
return hidden, cell10.1.2 Decoder
The decoder is a crucial component in neural machine translation models. It takes the context vector, which is produced by the encoder, and uses it to generate the output sequence. Generally, the decoder is also an RNN. The decoder accomplishes this task step-by-step, generating one token at each step. It utilizes the context vector and all of the tokens that have been generated up to that point as input.
It is important to note that the decoder is responsible for generating the output sequence, which is in turn responsible for translating the source language into the target language. Without the decoder, the neural machine translation model would be incomplete and unable to produce a translation. Therefore, the decoder plays a significant role in the success of the entire neural machine translation process.
Example:
Here's an example of a decoder implementation:
class Decoder(nn.Module):
def __init__(self, output_dim, emb_dim, hid_dim, n_layers, dropout):
super().__init__()
self.output_dim = output_dim
self.hid_dim = hid_dim
self.n_layers = n_layers
self.embedding = nn.Embedding(output_dim, emb_dim)
self.rnn = nn.LSTM(emb_dim, hid_dim, n_layers, dropout=dropout)
self.fc_out = nn.Linear(hid_dim, output_dim)
self.dropout = nn.Dropout(dropout)
def forward(self, input, hidden, cell):
input = input.unsqueeze(0)
embedded = self.dropout(self.embedding(input))
output, (hidden, cell) = self.rnn(embedded, (hidden, cell))
prediction = self.fc_out(output.squeeze(0))
return prediction, hidden, cell10.1.3 Seq2Seq Model
The Seq2Seq model is a powerful tool in natural language processing. It consists of two main components, namely the encoder and decoder, which work together to produce the desired output.
The encoder takes in the input sequence and produces a context vector, which captures the essential information needed for the decoding stage. In the decoding stage, the decoder uses this context vector to generate the output sequence, one step at a time.
It is important to note that the Seq2Seq model has many practical applications, including machine translation, text summarization, and speech recognition, among others. In fact, it is widely used in the industry due to its effectiveness and versatility.
Example:
Here's a simple implementation of a Seq2Seq model in PyTorch:
class Seq2Seq(nn.Module):
def __init__(self, encoder, decoder, device):
super().__init__()
self.encoder = encoder
self.decoder = decoder
self.device = device
def forward(self, src, trg, teacher_forcing_ratio=0.5):
batch_size = trg.shape[1]
trg_len = trg.shape[0]
trg_vocab_size = self.decoder.output_dim
outputs = torch.zeros(trg_len, batch_size, trg_vocab_size).to(self.device)
hidden, cell = self.encoder(src)
input = trg[0,:]
for t in range(1, trg_len):
output, hidden, cell = self.decoder(input, hidden, cell)
outputs[t] = output
teacher_force = random.random() < teacher_forcing_ratio
top1 = output.argmax(1)
input = trg[t] if teacher_force else top1
return outputsThis code snippet defines a Seq2Seq model that uses the previously defined encoder and decoder. The model uses a technique called teacher forcing during training. Teacher forcing is a strategy for training sequence to sequence models that uses the true output sequence as input to the decoder rather than the output generated by the decoder in the previous time step.