Code icon

The App is Under a Quick Maintenance

We apologize for the inconvenience. Please come back later

Cuantum Technologies logo
Dashboard
Perfil
Facturación
Full Access
Salir
Menu iconMenu iconNLP con Transformers, técnicas avanzadas y aplicaciones multimodales
NLP con Transformers, técnicas avanzadas y aplicaciones multimodales
Arrow Icon
Project 1: Machine Translation with MarianMT
Base Project Implementation
Step 1: Setting Up the Environment
Step 2: Loading the MarianMT Model
Step 3: Translating Text
Step 4: Exploring Additional Language Pairs
Advanced Customizations
Deploying a Translation API
Conclusion
Comprar libro
You've learned this already. ✅
Click here to view the next lesson.

Project 1: Machine Translation with MarianMT

Step 3: Translating Text

The translation process involves several sophisticated steps. First, the tokenizer converts your input text into numerical tokens that the model can understand. These tokens are then processed by the MarianMT model, which uses its trained neural network to generate the translation. The model employs attention mechanisms to understand context and maintain coherence in the translation. Finally, the output tokens are decoded back into human-readable text in the target language. This process happens automatically and efficiently, leveraging the power of modern deep learning architectures.

Preparing Multilingual Datasets

Before working with the dataset examples, ensure you have the pandas library installed:

pip install pandas

Pandas is a powerful data manipulation library for Python that makes it easy to load, process, and analyze structured data. It's particularly useful for handling large datasets and provides convenient functions for reading various file formats like CSV, Excel, and JSON.

For large-scale translation tasks, having access to comprehensive multilingual datasets is crucial for training and evaluating machine translation models. These datasets provide parallel texts in multiple languages, allowing for accurate translation learning and benchmarking. Here are some widely-used examples:

  1. TED Talks Dataset:
    • Link: TED Talks Translation Dataset (https://github.com/neulab/word-embeddings-for-nmt)
    • Contains transcripts of TED Talks in multiple languages, offering high-quality translations of contemporary speeches covering diverse topics from technology to social issues.
    • Particularly valuable for training conversational and presentation-style translation models.
    • Includes over 50,000 parallel sentences across multiple language pairs.
  2. Europarl Corpus:
    • Link: Europarl (https://www.statmt.org/europarl/)
    • A large corpus of parallel text from European Parliament proceedings, containing over 40 million words in each language.
    • Covers all official EU languages, making it ideal for European language translation tasks.
    • Known for its formal language and political terminology, perfect for training models on official document translation.

Here’s an example of loading and preprocessing a dataset:

import pandas as pd

# Load a sample multilingual dataset
df = pd.read_csv("path_to_dataset.csv")

# Filter for specific language pairs
en_text = df[df['lang'] == 'en']['text']
fr_text = df[df['lang'] == 'fr']['text']

# Prepare data for MarianMT
text_pairs = list(zip(en_text, fr_text))
print(f"Loaded {len(text_pairs)} sentence pairs for translation.")

Let's break down this code:

1. Data Loading

  • Uses pandas (pd) to read a CSV file containing the multilingual dataset
  • The CSV file is expected to have at least two columns: the text and language identifier ('lang')

2. Language Filtering

  • Filters the dataset to extract English texts: df[df['lang'] == 'en']['text']
  • Similarly extracts French texts: df[df['lang'] == 'fr']['text']

3. Data Preparation

  • Creates pairs of English-French translations using zip()
  • Stores these pairs in a list for easy access during translation

4. Status Report

  • Prints the total number of sentence pairs loaded, helping track the dataset size

Evaluating Translation Quality

To measure the quality of machine translation, we use several sophisticated evaluation metrics:

BLEU (Bilingual Evaluation Understudy) compares the machine-generated translation against one or more reference translations by calculating n-gram overlap. It produces a score between 0 and 1, where higher scores indicate better translations. BLEU is particularly good at measuring translation accuracy at the phrase level.

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) focuses on comparing overlapping units such as n-grams, word sequences, and word pairs between the machine translation and reference texts. It's especially useful for evaluating the fluency and readability of translations.

BERTScore leverages contextual embeddings from BERT to compute similarity scores by matching words in candidate and reference translations. This metric better captures semantic similarities even when exact words don't match, making it particularly effective for evaluating meaning preservation in translations.

These metrics work together to provide a comprehensive assessment of translation quality, helping evaluate both linguistic accuracy and semantic faithfulness to the original text.

Here’s an example of using the nltk library to calculate the BLEU score:

pip install nltk
from nltk.translate.bleu_score import sentence_bleu, SmoothingFunction

def evaluate_bleu(reference, hypothesis):
    """
    Calculate BLEU score for a single sentence.
    Args:
        reference (list): List of reference translations (tokenized).
        hypothesis (list): Hypothesis translation (tokenized).
    Returns:
        float: BLEU score
    """
    smoothing_function = SmoothingFunction().method4
    bleu_score = sentence_bleu([reference], hypothesis, smoothing_function=smoothing_function)
    return bleu_score

# Example usage
reference = "Bonjour, comment ça va ?".split()  # Reference translation (tokenized)
hypothesis = "Salut, comment ça va ?".split()  # Model's output (tokenized)

bleu = evaluate_bleu(reference, hypothesis)
print(f"BLEU Score: {bleu:.2f}")

Here's a breakdown of the code:

1. Function Definition:

  • The evaluate_bleu function takes two parameters:
    • reference: The correct translation (as a list of tokens)
    • hypothesis: The machine-generated translation (as a list of tokens)

2. Smoothing Function:

  • Uses NLTK's SmoothingFunction.method4 to handle cases where there are no matching n-grams between the reference and hypothesis
    This prevents the BLEU score from being zero due to no matches

3. BLEU Score Calculation:

  • The function calculates a score between 0 and 1, where higher scores indicate better translations
  • Uses sentence_bleu() from NLTK to compare the overlap between the reference and hypothesis translations

4. Example Usage:

  • Shows a practical example using French phrases:
    • Reference translation: "Bonjour, comment ça va ?"
    • Hypothesis translation: "Salut, comment ça va ?"
      The code splits these strings into tokens and calculates their BLEU score

This evaluation metric is particularly effective at measuring translation accuracy at the phrase level, making it a valuable tool for assessing machine translation quality.

Step 3: Translating Text

The translation process involves several sophisticated steps. First, the tokenizer converts your input text into numerical tokens that the model can understand. These tokens are then processed by the MarianMT model, which uses its trained neural network to generate the translation. The model employs attention mechanisms to understand context and maintain coherence in the translation. Finally, the output tokens are decoded back into human-readable text in the target language. This process happens automatically and efficiently, leveraging the power of modern deep learning architectures.

Preparing Multilingual Datasets

Before working with the dataset examples, ensure you have the pandas library installed:

pip install pandas

Pandas is a powerful data manipulation library for Python that makes it easy to load, process, and analyze structured data. It's particularly useful for handling large datasets and provides convenient functions for reading various file formats like CSV, Excel, and JSON.

For large-scale translation tasks, having access to comprehensive multilingual datasets is crucial for training and evaluating machine translation models. These datasets provide parallel texts in multiple languages, allowing for accurate translation learning and benchmarking. Here are some widely-used examples:

  1. TED Talks Dataset:
    • Link: TED Talks Translation Dataset (https://github.com/neulab/word-embeddings-for-nmt)
    • Contains transcripts of TED Talks in multiple languages, offering high-quality translations of contemporary speeches covering diverse topics from technology to social issues.
    • Particularly valuable for training conversational and presentation-style translation models.
    • Includes over 50,000 parallel sentences across multiple language pairs.
  2. Europarl Corpus:
    • Link: Europarl (https://www.statmt.org/europarl/)
    • A large corpus of parallel text from European Parliament proceedings, containing over 40 million words in each language.
    • Covers all official EU languages, making it ideal for European language translation tasks.
    • Known for its formal language and political terminology, perfect for training models on official document translation.

Here’s an example of loading and preprocessing a dataset:

import pandas as pd

# Load a sample multilingual dataset
df = pd.read_csv("path_to_dataset.csv")

# Filter for specific language pairs
en_text = df[df['lang'] == 'en']['text']
fr_text = df[df['lang'] == 'fr']['text']

# Prepare data for MarianMT
text_pairs = list(zip(en_text, fr_text))
print(f"Loaded {len(text_pairs)} sentence pairs for translation.")

Let's break down this code:

1. Data Loading

  • Uses pandas (pd) to read a CSV file containing the multilingual dataset
  • The CSV file is expected to have at least two columns: the text and language identifier ('lang')

2. Language Filtering

  • Filters the dataset to extract English texts: df[df['lang'] == 'en']['text']
  • Similarly extracts French texts: df[df['lang'] == 'fr']['text']

3. Data Preparation

  • Creates pairs of English-French translations using zip()
  • Stores these pairs in a list for easy access during translation

4. Status Report

  • Prints the total number of sentence pairs loaded, helping track the dataset size

Evaluating Translation Quality

To measure the quality of machine translation, we use several sophisticated evaluation metrics:

BLEU (Bilingual Evaluation Understudy) compares the machine-generated translation against one or more reference translations by calculating n-gram overlap. It produces a score between 0 and 1, where higher scores indicate better translations. BLEU is particularly good at measuring translation accuracy at the phrase level.

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) focuses on comparing overlapping units such as n-grams, word sequences, and word pairs between the machine translation and reference texts. It's especially useful for evaluating the fluency and readability of translations.

BERTScore leverages contextual embeddings from BERT to compute similarity scores by matching words in candidate and reference translations. This metric better captures semantic similarities even when exact words don't match, making it particularly effective for evaluating meaning preservation in translations.

These metrics work together to provide a comprehensive assessment of translation quality, helping evaluate both linguistic accuracy and semantic faithfulness to the original text.

Here’s an example of using the nltk library to calculate the BLEU score:

pip install nltk
from nltk.translate.bleu_score import sentence_bleu, SmoothingFunction

def evaluate_bleu(reference, hypothesis):
    """
    Calculate BLEU score for a single sentence.
    Args:
        reference (list): List of reference translations (tokenized).
        hypothesis (list): Hypothesis translation (tokenized).
    Returns:
        float: BLEU score
    """
    smoothing_function = SmoothingFunction().method4
    bleu_score = sentence_bleu([reference], hypothesis, smoothing_function=smoothing_function)
    return bleu_score

# Example usage
reference = "Bonjour, comment ça va ?".split()  # Reference translation (tokenized)
hypothesis = "Salut, comment ça va ?".split()  # Model's output (tokenized)

bleu = evaluate_bleu(reference, hypothesis)
print(f"BLEU Score: {bleu:.2f}")

Here's a breakdown of the code:

1. Function Definition:

  • The evaluate_bleu function takes two parameters:
    • reference: The correct translation (as a list of tokens)
    • hypothesis: The machine-generated translation (as a list of tokens)

2. Smoothing Function:

  • Uses NLTK's SmoothingFunction.method4 to handle cases where there are no matching n-grams between the reference and hypothesis
    This prevents the BLEU score from being zero due to no matches

3. BLEU Score Calculation:

  • The function calculates a score between 0 and 1, where higher scores indicate better translations
  • Uses sentence_bleu() from NLTK to compare the overlap between the reference and hypothesis translations

4. Example Usage:

  • Shows a practical example using French phrases:
    • Reference translation: "Bonjour, comment ça va ?"
    • Hypothesis translation: "Salut, comment ça va ?"
      The code splits these strings into tokens and calculates their BLEU score

This evaluation metric is particularly effective at measuring translation accuracy at the phrase level, making it a valuable tool for assessing machine translation quality.

Step 3: Translating Text

The translation process involves several sophisticated steps. First, the tokenizer converts your input text into numerical tokens that the model can understand. These tokens are then processed by the MarianMT model, which uses its trained neural network to generate the translation. The model employs attention mechanisms to understand context and maintain coherence in the translation. Finally, the output tokens are decoded back into human-readable text in the target language. This process happens automatically and efficiently, leveraging the power of modern deep learning architectures.

Preparing Multilingual Datasets

Before working with the dataset examples, ensure you have the pandas library installed:

pip install pandas

Pandas is a powerful data manipulation library for Python that makes it easy to load, process, and analyze structured data. It's particularly useful for handling large datasets and provides convenient functions for reading various file formats like CSV, Excel, and JSON.

For large-scale translation tasks, having access to comprehensive multilingual datasets is crucial for training and evaluating machine translation models. These datasets provide parallel texts in multiple languages, allowing for accurate translation learning and benchmarking. Here are some widely-used examples:

  1. TED Talks Dataset:
    • Link: TED Talks Translation Dataset (https://github.com/neulab/word-embeddings-for-nmt)
    • Contains transcripts of TED Talks in multiple languages, offering high-quality translations of contemporary speeches covering diverse topics from technology to social issues.
    • Particularly valuable for training conversational and presentation-style translation models.
    • Includes over 50,000 parallel sentences across multiple language pairs.
  2. Europarl Corpus:
    • Link: Europarl (https://www.statmt.org/europarl/)
    • A large corpus of parallel text from European Parliament proceedings, containing over 40 million words in each language.
    • Covers all official EU languages, making it ideal for European language translation tasks.
    • Known for its formal language and political terminology, perfect for training models on official document translation.

Here’s an example of loading and preprocessing a dataset:

import pandas as pd

# Load a sample multilingual dataset
df = pd.read_csv("path_to_dataset.csv")

# Filter for specific language pairs
en_text = df[df['lang'] == 'en']['text']
fr_text = df[df['lang'] == 'fr']['text']

# Prepare data for MarianMT
text_pairs = list(zip(en_text, fr_text))
print(f"Loaded {len(text_pairs)} sentence pairs for translation.")

Let's break down this code:

1. Data Loading

  • Uses pandas (pd) to read a CSV file containing the multilingual dataset
  • The CSV file is expected to have at least two columns: the text and language identifier ('lang')

2. Language Filtering

  • Filters the dataset to extract English texts: df[df['lang'] == 'en']['text']
  • Similarly extracts French texts: df[df['lang'] == 'fr']['text']

3. Data Preparation

  • Creates pairs of English-French translations using zip()
  • Stores these pairs in a list for easy access during translation

4. Status Report

  • Prints the total number of sentence pairs loaded, helping track the dataset size

Evaluating Translation Quality

To measure the quality of machine translation, we use several sophisticated evaluation metrics:

BLEU (Bilingual Evaluation Understudy) compares the machine-generated translation against one or more reference translations by calculating n-gram overlap. It produces a score between 0 and 1, where higher scores indicate better translations. BLEU is particularly good at measuring translation accuracy at the phrase level.

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) focuses on comparing overlapping units such as n-grams, word sequences, and word pairs between the machine translation and reference texts. It's especially useful for evaluating the fluency and readability of translations.

BERTScore leverages contextual embeddings from BERT to compute similarity scores by matching words in candidate and reference translations. This metric better captures semantic similarities even when exact words don't match, making it particularly effective for evaluating meaning preservation in translations.

These metrics work together to provide a comprehensive assessment of translation quality, helping evaluate both linguistic accuracy and semantic faithfulness to the original text.

Here’s an example of using the nltk library to calculate the BLEU score:

pip install nltk
from nltk.translate.bleu_score import sentence_bleu, SmoothingFunction

def evaluate_bleu(reference, hypothesis):
    """
    Calculate BLEU score for a single sentence.
    Args:
        reference (list): List of reference translations (tokenized).
        hypothesis (list): Hypothesis translation (tokenized).
    Returns:
        float: BLEU score
    """
    smoothing_function = SmoothingFunction().method4
    bleu_score = sentence_bleu([reference], hypothesis, smoothing_function=smoothing_function)
    return bleu_score

# Example usage
reference = "Bonjour, comment ça va ?".split()  # Reference translation (tokenized)
hypothesis = "Salut, comment ça va ?".split()  # Model's output (tokenized)

bleu = evaluate_bleu(reference, hypothesis)
print(f"BLEU Score: {bleu:.2f}")

Here's a breakdown of the code:

1. Function Definition:

  • The evaluate_bleu function takes two parameters:
    • reference: The correct translation (as a list of tokens)
    • hypothesis: The machine-generated translation (as a list of tokens)

2. Smoothing Function:

  • Uses NLTK's SmoothingFunction.method4 to handle cases where there are no matching n-grams between the reference and hypothesis
    This prevents the BLEU score from being zero due to no matches

3. BLEU Score Calculation:

  • The function calculates a score between 0 and 1, where higher scores indicate better translations
  • Uses sentence_bleu() from NLTK to compare the overlap between the reference and hypothesis translations

4. Example Usage:

  • Shows a practical example using French phrases:
    • Reference translation: "Bonjour, comment ça va ?"
    • Hypothesis translation: "Salut, comment ça va ?"
      The code splits these strings into tokens and calculates their BLEU score

This evaluation metric is particularly effective at measuring translation accuracy at the phrase level, making it a valuable tool for assessing machine translation quality.

Step 3: Translating Text

The translation process involves several sophisticated steps. First, the tokenizer converts your input text into numerical tokens that the model can understand. These tokens are then processed by the MarianMT model, which uses its trained neural network to generate the translation. The model employs attention mechanisms to understand context and maintain coherence in the translation. Finally, the output tokens are decoded back into human-readable text in the target language. This process happens automatically and efficiently, leveraging the power of modern deep learning architectures.

Preparing Multilingual Datasets

Before working with the dataset examples, ensure you have the pandas library installed:

pip install pandas

Pandas is a powerful data manipulation library for Python that makes it easy to load, process, and analyze structured data. It's particularly useful for handling large datasets and provides convenient functions for reading various file formats like CSV, Excel, and JSON.

For large-scale translation tasks, having access to comprehensive multilingual datasets is crucial for training and evaluating machine translation models. These datasets provide parallel texts in multiple languages, allowing for accurate translation learning and benchmarking. Here are some widely-used examples:

  1. TED Talks Dataset:
    • Link: TED Talks Translation Dataset (https://github.com/neulab/word-embeddings-for-nmt)
    • Contains transcripts of TED Talks in multiple languages, offering high-quality translations of contemporary speeches covering diverse topics from technology to social issues.
    • Particularly valuable for training conversational and presentation-style translation models.
    • Includes over 50,000 parallel sentences across multiple language pairs.
  2. Europarl Corpus:
    • Link: Europarl (https://www.statmt.org/europarl/)
    • A large corpus of parallel text from European Parliament proceedings, containing over 40 million words in each language.
    • Covers all official EU languages, making it ideal for European language translation tasks.
    • Known for its formal language and political terminology, perfect for training models on official document translation.

Here’s an example of loading and preprocessing a dataset:

import pandas as pd

# Load a sample multilingual dataset
df = pd.read_csv("path_to_dataset.csv")

# Filter for specific language pairs
en_text = df[df['lang'] == 'en']['text']
fr_text = df[df['lang'] == 'fr']['text']

# Prepare data for MarianMT
text_pairs = list(zip(en_text, fr_text))
print(f"Loaded {len(text_pairs)} sentence pairs for translation.")

Let's break down this code:

1. Data Loading

  • Uses pandas (pd) to read a CSV file containing the multilingual dataset
  • The CSV file is expected to have at least two columns: the text and language identifier ('lang')

2. Language Filtering

  • Filters the dataset to extract English texts: df[df['lang'] == 'en']['text']
  • Similarly extracts French texts: df[df['lang'] == 'fr']['text']

3. Data Preparation

  • Creates pairs of English-French translations using zip()
  • Stores these pairs in a list for easy access during translation

4. Status Report

  • Prints the total number of sentence pairs loaded, helping track the dataset size

Evaluating Translation Quality

To measure the quality of machine translation, we use several sophisticated evaluation metrics:

BLEU (Bilingual Evaluation Understudy) compares the machine-generated translation against one or more reference translations by calculating n-gram overlap. It produces a score between 0 and 1, where higher scores indicate better translations. BLEU is particularly good at measuring translation accuracy at the phrase level.

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) focuses on comparing overlapping units such as n-grams, word sequences, and word pairs between the machine translation and reference texts. It's especially useful for evaluating the fluency and readability of translations.

BERTScore leverages contextual embeddings from BERT to compute similarity scores by matching words in candidate and reference translations. This metric better captures semantic similarities even when exact words don't match, making it particularly effective for evaluating meaning preservation in translations.

These metrics work together to provide a comprehensive assessment of translation quality, helping evaluate both linguistic accuracy and semantic faithfulness to the original text.

Here’s an example of using the nltk library to calculate the BLEU score:

pip install nltk
from nltk.translate.bleu_score import sentence_bleu, SmoothingFunction

def evaluate_bleu(reference, hypothesis):
    """
    Calculate BLEU score for a single sentence.
    Args:
        reference (list): List of reference translations (tokenized).
        hypothesis (list): Hypothesis translation (tokenized).
    Returns:
        float: BLEU score
    """
    smoothing_function = SmoothingFunction().method4
    bleu_score = sentence_bleu([reference], hypothesis, smoothing_function=smoothing_function)
    return bleu_score

# Example usage
reference = "Bonjour, comment ça va ?".split()  # Reference translation (tokenized)
hypothesis = "Salut, comment ça va ?".split()  # Model's output (tokenized)

bleu = evaluate_bleu(reference, hypothesis)
print(f"BLEU Score: {bleu:.2f}")

Here's a breakdown of the code:

1. Function Definition:

  • The evaluate_bleu function takes two parameters:
    • reference: The correct translation (as a list of tokens)
    • hypothesis: The machine-generated translation (as a list of tokens)

2. Smoothing Function:

  • Uses NLTK's SmoothingFunction.method4 to handle cases where there are no matching n-grams between the reference and hypothesis
    This prevents the BLEU score from being zero due to no matches

3. BLEU Score Calculation:

  • The function calculates a score between 0 and 1, where higher scores indicate better translations
  • Uses sentence_bleu() from NLTK to compare the overlap between the reference and hypothesis translations

4. Example Usage:

  • Shows a practical example using French phrases:
    • Reference translation: "Bonjour, comment ça va ?"
    • Hypothesis translation: "Salut, comment ça va ?"
      The code splits these strings into tokens and calculates their BLEU score

This evaluation metric is particularly effective at measuring translation accuracy at the phrase level, making it a valuable tool for assessing machine translation quality.

Compra este libro