Code icon

The App is Under a Quick Maintenance

We apologize for the inconvenience. Please come back later

Cuantum Technologies logo
Dashboard
Profile
Billing
Full Access
Log out
Menu iconMenu iconDeep Learning and AI Superhero
Deep Learning and AI Superhero
Arrow Icon
Chapter 9: Practical Projects
9.1 Project 1: Predicting House Prices with Regression
9.2 Project 2: Sentiment Analysis Using Transformer-based Models
9.3 Project 3: Image Classification with CNNs
9.4 Project 4: Time Series Forecasting with LSTMs (Improved)
9.5 Project 5: GAN-based Image Generation
Buy this book
You've learned this already. ✅
Click here to view the next lesson.

Chapter 9: Practical Projects

9.2 Project 2: Sentiment Analysis Using Transformer-based Models

Sentiment analysis is a fundamental and highly significant task within the field of Natural Language Processing (NLP), focusing on the intricate process of deciphering and interpreting the underlying emotional tone or attitude expressed in a given piece of text.

This project delves into the application of cutting-edge transformer-based models, with a particular emphasis on BERT (Bidirectional Encoder Representations from Transformers), to conduct sophisticated sentiment analysis on textual data.

By leveraging these advanced neural network architectures, we aim to develop a robust system capable of accurately discerning and categorizing the sentiments conveyed in various forms of written communication, ranging from social media posts and product reviews to news articles and beyond.

9.2.1 Problem Statement and Dataset

For this project, we will utilize the IMDB Movie Reviews dataset, a comprehensive collection comprising 50,000 movie reviews. Each review in this dataset has been meticulously labeled as either positive or negative, providing a rich source of sentiment-annotated data.

Our primary objective is to develop and train a sophisticated model capable of accurately discerning and classifying the underlying sentiment expressed in these movie reviews. This task presents an excellent opportunity to apply advanced natural language processing techniques to real-world textual data, with the ultimate aim of creating a robust sentiment analysis system that can effectively interpret the nuanced opinions and emotions conveyed in written film critiques.

Loading and Exploring the Dataset

import pandas as pd
from datasets import load_dataset
from sklearn.model_selection import train_test_split

# Load the IMDB dataset
dataset = load_dataset('imdb')

# Convert to pandas DataFrame for easier manipulation
train_df = pd.DataFrame(dataset['train'])
test_df = pd.DataFrame(dataset['test'])

# Display basic information about the dataset
print(train_df.info())
print(train_df['label'].value_counts(normalize=True))

# Display a few examples
print(train_df.head())

Here's a breakdown of what it does:

  • Imports necessary libraries: pandas for data manipulation, load_dataset from the datasets library to load the IMDB dataset, and train_test_split from sklearn for data splitting (although not used in this snippet)
  • Loads the IMDB dataset using load_dataset('imdb')
  • Converts the train and test sets to pandas DataFrames for easier manipulation
  • Displays basic information about the training dataset using train_df.info()
  • Shows the distribution of labels (positive/negative sentiment) in the training set using train_df['label'].value_counts(normalize=True)
  • Displays the first few examples from the training set using train_df.head()

9.2.2 Data Preprocessing

Before we can feed our data into the BERT model for analysis, it is crucial to undergo a comprehensive preprocessing stage. This essential step involves several key processes that prepare the raw text data for optimal processing by our advanced neural network.

The primary components of this preprocessing phase include tokenization, which breaks down the text into individual units or tokens that the model can interpret; padding, which ensures all input sequences are of uniform length for batch processing; and the creation of attention masks, which guide the model's focus to relevant parts of the input while ignoring padding tokens. 

These preprocessing steps are fundamental in transforming our raw textual data into a format that can be efficiently and effectively processed by our BERT model, ultimately enabling more accurate sentiment analysis results.

from transformers import BertTokenizer
import torch

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

def preprocess_data(texts, labels, max_length=256):
    encoded = tokenizer.batch_encode_plus(
        texts,
        add_special_tokens=True,
        max_length=max_length,
        padding='max_length',
        truncation=True,
        return_attention_mask=True,
        return_tensors='pt'
    )
    return {
        'input_ids': encoded['input_ids'],
        'attention_mask': encoded['attention_mask'],
        'labels': torch.tensor(labels)
    }

# Preprocess the data
train_data = preprocess_data(train_df['text'].tolist(), train_df['label'].tolist())
test_data = preprocess_data(test_df['text'].tolist(), test_df['label'].tolist())

Here's a breakdown of what the code does:

  • It imports the necessary libraries: BertTokenizer from transformers and torch
  • It initializes a BERT tokenizer using the 'bert-base-uncased' model
  • The preprocess_data function is defined, which takes texts and labels as input, along with an optional max_length parameter
  • Inside the function, it uses the tokenizer's batch_encode_plus method to encode the input texts. This method:
    • Adds special tokens (like [CLS] and [SEP])
    • Pads or truncates sequences to a maximum length
    • Creates attention masks
    • Returns tensors suitable for PyTorch
  • The function returns a dictionary containing:
    • input_ids: the encoded and padded text sequences
    • attention_mask: a mask indicating which tokens are padding (0) and which are not (1)
    • labels: the sentiment labels converted to a PyTorch tensor
  • Finally, the code applies this preprocessing function to both the training and testing data, creating train_data and test_data

This preprocessing step is crucial as it transforms the raw text data into a format that can be efficiently processed by the BERT model for sentiment analysis

9.2.3 Building and Training the BERT Model

For this project, we will harness the power of the BertForSequenceClassification model, a sophisticated tool available in the transformers library. This advanced model is meticulously engineered and optimized for text classification tasks, making it an ideal choice for our sentiment analysis endeavor.

By leveraging this state-of-the-art architecture, we can effectively capture the nuanced sentiments expressed in our movie review dataset, enabling highly accurate classification of positive and negative sentiments.

from transformers import BertForSequenceClassification, AdamW
from torch.utils.data import DataLoader, TensorDataset
import torch.nn as nn

# Set up the model
model = BertForSequenceClassification.from_pretrained('bert-base-uncased', num_labels=2)

# Set up the optimizer
optimizer = AdamW(model.parameters(), lr=2e-5)

# Create DataLoader
train_dataset = TensorDataset(train_data['input_ids'], train_data['attention_mask'], train_data['labels'])
train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True)

# Training loop
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model.to(device)

num_epochs = 3
for epoch in range(num_epochs):
    model.train()
    for batch in train_loader:
        input_ids, attention_mask, labels = [b.to(device) for b in batch]
        
        optimizer.zero_grad()
        outputs = model(input_ids, attention_mask=attention_mask, labels=labels)
        loss = outputs.loss
        loss.backward()
        optimizer.step()
    
    print(f"Epoch {epoch+1}/{num_epochs} completed")

# Save the model
torch.save(model.state_dict(), 'bert_sentiment_model.pth')

Here's a breakdown of the main components:

  1. Model Setup: The code initializes a BertForSequenceClassification model, which is pre-trained and fine-tuned for sequence classification tasks like sentiment analysis.
  2. Optimizer: It sets up an AdamW optimizer, which is an improved version of Adam, commonly used for training deep learning models.
  3. Data Preparation: The code creates a TensorDataset and a DataLoader to efficiently batch and shuffle the training data.
  4. Training Loop: The model is trained for 3 epochs. In each epoch:
    • - It iterates through batches of data
    • - Computes the loss
    • - Performs backpropagation
    • - Updates the model parameters
  5. Device Utilization: The code checks for GPU availability and moves the model to the appropriate device (CPU or GPU) for efficient computation.
  6. Model Saving: After training, the model's state dictionary is saved to a file for future use.

This implementation allows for effective training of a BERT model on the IMDB movie review dataset, enabling it to learn and classify the sentiment (positive or negative) of movie reviews.

9.2.4 Evaluating the Model

After completing the training phase, it is crucial to assess our model's effectiveness and accuracy by evaluating its performance on the test set. This evaluation step allows us to gauge how well our BERT-based sentiment analysis model generalizes to unseen data and provides valuable insights into its real-world applicability.

By analyzing various metrics such as accuracy, precision, recall, and F1-score, we can gain a comprehensive understanding of our model's strengths and potential areas for improvement.

from sklearn.metrics import accuracy_score, classification_report

model.eval()
test_dataset = TensorDataset(test_data['input_ids'], test_data['attention_mask'], test_data['labels'])
test_loader = DataLoader(test_dataset, batch_size=32)

all_preds = []
all_labels = []

with torch.no_grad():
    for batch in test_loader:
        input_ids, attention_mask, labels = [b.to(device) for b in batch]
        outputs = model(input_ids, attention_mask=attention_mask)
        preds = torch.argmax(outputs.logits, dim=1)
        all_preds.extend(preds.cpu().numpy())
        all_labels.extend(labels.cpu().numpy())

accuracy = accuracy_score(all_labels, all_preds)
print(f"Accuracy: {accuracy}")
print(classification_report(all_labels, all_preds))

Here's a breakdown of what it does:

  • It imports necessary metrics from sklearn for model evaluation
  • Sets the model to evaluation mode with model.eval()
  • Creates a TensorDataset and DataLoader for the test data, which helps in batch processing
  • Initializes empty lists to store all predictions and true labels
  • Uses a with torch.no_grad() context to disable gradient calculations during inference, which saves memory and speeds up computation
  • Iterates through the test data in batches:
    • Moves the input data to the appropriate device (CPU or GPU)
    • Generates predictions using the model
    • Extracts the predicted class (sentiment) for each sample
    • Adds the predictions and true labels to their respective lists
  • Calculates the overall accuracy of the model using accuracy_score
  • Prints a detailed classification report, which typically includes precision, recall, and F1-score for each class

This evaluation process allows us to assess how well the model performs on unseen data, giving us insights into its effectiveness for sentiment analysis tasks.

9.2.5 Inference with New Text

With our model now fully trained and optimized, we can harness its capabilities to analyze and predict the sentiment of new, previously unseen text. This practical application of our sentiment analysis model allows us to gain valuable insights from fresh, real-world data, demonstrating the model's effectiveness beyond the training dataset.

By leveraging the power of our fine-tuned BERT-based model, we can now confidently assess the emotional tone of various text inputs, ranging from customer reviews and social media posts to news articles and beyond, providing a robust tool for understanding public opinion and consumer sentiment across diverse domains.

def predict_sentiment(text):
    encoded = tokenizer.encode_plus(
        text,
        add_special_tokens=True,
        max_length=256,
        padding='max_length',
        truncation=True,
        return_attention_mask=True,
        return_tensors='pt'
    )
    
    input_ids = encoded['input_ids'].to(device)
    attention_mask = encoded['attention_mask'].to(device)
    
    with torch.no_grad():
        outputs = model(input_ids, attention_mask=attention_mask)
        pred = torch.argmax(outputs.logits, dim=1)
    
    return "Positive" if pred.item() == 1 else "Negative"

# Example usage
new_review = "This movie was absolutely fantastic! I loved every minute of it."
sentiment = predict_sentiment(new_review)
print(f"Predicted sentiment: {sentiment}")

Here's a breakdown of what the code does:

  1. The function takes a text input and preprocesses it using the BERT tokenizer
  2. It encodes the text, adding special tokens, padding, and creating an attention mask
  3. The encoded input is then passed through the BERT model to get predictions
  4. The model's output logits are used to determine the sentiment (positive or negative)
  5. The function returns "Positive" if the prediction is 1, and "Negative" otherwise

The code also includes an example usage of the function:

  1. A sample review is provided: "This movie was absolutely fantastic! I loved every minute of it."
  2. The predict_sentiment function is called with this review
  3. The predicted sentiment is then printed

This function allows for easy sentiment analysis of new, unseen text using the trained BERT model, demonstrating its practical application for analyzing various text inputs such as customer reviews or social media posts

9.2.6 Advanced Techniques

Fine-tuning with Discriminative Learning Rates

To enhance our model's performance, we can implement discriminative learning rates, a sophisticated technique where different components of the model are trained at varying rates. This approach allows for more nuanced optimization, as it recognizes that different layers of the neural network may require different learning paces.

By applying higher learning rates to the upper layers of the model, which are more task-specific, and lower rates to the lower layers, which capture more general features, we can fine-tune the model more effectively.

This method is particularly beneficial when working with pre-trained models like BERT, as it allows us to carefully adjust the model's parameters without disrupting the valuable information learned during pre-training.

from transformers import get_linear_schedule_with_warmup

# Prepare optimizer and schedule (linear warmup and decay)
no_decay = ['bias', 'LayerNorm.weight']
optimizer_grouped_parameters = [
    {'params': [p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)], 'weight_decay': 0.01},
    {'params': [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)], 'weight_decay': 0.0}
]

optimizer = AdamW(optimizer_grouped_parameters, lr=2e-5, eps=1e-8)
scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps=0, num_training_steps=len(train_loader) * num_epochs)

# Update the training loop to use the scheduler
for epoch in range(num_epochs):
    model.train()
    for batch in train_loader:
        input_ids, attention_mask, labels = [b.to(device) for b in batch]
        
        optimizer.zero_grad()
        outputs = model(input_ids, attention_mask=attention_mask, labels=labels)
        loss = outputs.loss
        loss.backward()
        optimizer.step()
        scheduler.step()
    
    print(f"Epoch {epoch+1}/{num_epochs} completed")

Here's a breakdown of the key components:

  1. Importing the scheduler: The code imports get_linear_schedule_with_warmup from the transformers library.
  2. Optimizer configuration: 
    • - It sets up two groups of parameters: one with weight decay and one without.
    • - This approach helps in applying different learning rates to different parts of the model.
  3. Optimizer and scheduler initialization: 
    • - The AdamW optimizer is initialized with the grouped parameters.
    • - A linear learning rate scheduler with warmup is created, which will adjust the learning rate during training.
  4. Training loop: 
    • - The code updates the training loop to incorporate the scheduler.
    • - After each optimization step, the scheduler is stepped to adjust the learning rate.

This implementation allows for more effective fine-tuning of the BERT model by applying different learning rates to different parts of the model and gradually adjusting the learning rate throughout the training process.

Data Augmentation

To enhance our dataset and potentially improve model performance, we can employ various data augmentation techniques. One particularly effective method is back-translation, which involves translating the original text to another language and then back to the original language. This process introduces subtle variations in the text while preserving its overall meaning and sentiment.

Additionally, we can explore other augmentation strategies such as synonym replacement, random insertion or deletion of words, and text paraphrasing. These techniques collectively help to increase the diversity and size of our training data, potentially leading to a more robust and generalizable sentiment analysis model.

from transformers import MarianMTModel, MarianTokenizer

# Load translation models
en_to_fr = MarianMTModel.from_pretrained('Helsinki-NLP/opus-mt-en-fr')
fr_to_en = MarianMTModel.from_pretrained('Helsinki-NLP/opus-mt-fr-en')
en_tokenizer = MarianTokenizer.from_pretrained('Helsinki-NLP/opus-mt-en-fr')
fr_tokenizer = MarianTokenizer.from_pretrained('Helsinki-NLP/opus-mt-fr-en')

def back_translate(text):
    # Translate to French
    fr_text = en_to_fr.generate(**en_tokenizer(text, return_tensors="pt", padding=True))
    fr_text = [en_tokenizer.decode(t, skip_special_tokens=True) for t in fr_text][0]
    
    # Translate back to English
    en_text = fr_to_en.generate(**fr_tokenizer(fr_text, return_tensors="pt", padding=True))
    en_text = [fr_tokenizer.decode(t, skip_special_tokens=True) for t in en_text][0]
    
    return en_text

# Augment the training data
augmented_texts = [back_translate(text) for text in train_df['text'][:1000]]  # Augment first 1000 samples
augmented_labels = train_df['label'][:1000]

train_df = pd.concat([train_df, pd.DataFrame({'text': augmented_texts, 'label': augmented_labels})])

Here's an explanation of the key components:

  • The code imports necessary models and tokenizers from the Transformers library for translation tasks.
  • It loads pre-trained models for English-to-French and French-to-English translation.
  • The back_translate function is defined to perform the augmentation:
    • It translates the input English text to French
    • Then translates the French text back to English
    • This process introduces subtle variations while preserving the overall meaning
  • The code then augments the training data:
    • It applies back-translation to the first 1000 samples of the training data
    • The augmented texts and their corresponding labels are added to the training dataset

This technique helps increase the diversity of the training data, potentially leading to a more robust and generalizable sentiment analysis model.

Ensemble Methods

To potentially enhance our model's performance and robustness, we can implement an ensemble approach. This technique involves creating multiple models, each with its own strengths and characteristics, and combining their predictions to generate a more accurate and reliable final output.

By leveraging the collective intelligence of various models, we can often achieve better results than relying on a single model alone. This ensemble method can help mitigate individual model weaknesses and capture a broader range of patterns in the data, ultimately leading to improved sentiment analysis accuracy.

# Train multiple models (e.g., BERT, RoBERTa, DistilBERT)
from transformers import RobertaForSequenceClassification, DistilBertForSequenceClassification

models = [
    BertForSequenceClassification.from_pretrained('bert-base-uncased', num_labels=2),
    RobertaForSequenceClassification.from_pretrained('roberta-base', num_labels=2),
    DistilBertForSequenceClassification.from_pretrained('distilbert-base-uncased', num_labels=2)
]

# Train each model (code omitted for brevity)

def ensemble_predict(text):
    encoded = tokenizer.encode_plus(
        text,
        add_special_tokens=True,
        max_length=256,
        padding='max_length',
        truncation=True,
        return_attention_mask=True,
        return_tensors='pt'
    )
    
    input_ids = encoded['input_ids'].to(device)
    attention_mask = encoded['attention_mask'].to(device)
    
    predictions = []
    with torch.no_grad():
        for model in models:
            outputs = model(input_ids, attention_mask=attention_mask)
            pred = torch.softmax(outputs.logits, dim=1)
            predictions.append(pred)
    
    # Average predictions
    avg_pred = torch.mean(torch.stack(predictions), dim=0)
    final_pred = torch.argmax(avg_pred, dim=1)
    
    return "Positive" if final_pred.item() == 1 else "Negative"

This code implements an ensemble method for sentiment analysis using multiple transformer-based models. Here's a breakdown of its key components:

  1. Model Initialization: The code imports and initializes three different pre-trained models: BERT, RoBERTa, and DistilBERT. Each model is set up for binary classification (positive/negative sentiment).
  2. Ensemble Prediction Function: The ensemble_predict function is defined to make predictions using all three models:
    • It tokenizes and encodes the input text using a tokenizer (presumably BERT's tokenizer, though this isn't explicitly shown in the snippet).
    • The encoded input is then passed through each model to get predictions.
    • The raw logits from each model are converted to probabilities using softmax.
    • The predictions from all models are averaged to get a final prediction.
    • The function returns "Positive" or "Negative" based on the averaged prediction.

This ensemble approach aims to improve prediction accuracy by combining the strengths of multiple models, potentially leading to more robust sentiment analysis results.

9.2.7 Conclusion

In this enhanced project, we have successfully implemented a sophisticated sentiment analysis model leveraging the power of BERT architecture. We've delved into advanced techniques to significantly boost its performance and versatility. Our comprehensive approach covered crucial aspects including meticulous data preprocessing, rigorous model training, thorough evaluation, and seamless inference processes.

Furthermore, we've introduced and explored cutting-edge techniques to push the boundaries of model performance. These include the implementation of discriminative learning rates, which allow for fine-tuned optimization across different layers of the model. We've also incorporated data augmentation strategies, particularly back-translation, to enrich our dataset and improve the model's ability to generalize. Additionally, we've ventured into ensemble methods, combining the strengths of multiple models to achieve more robust and accurate predictions.

These advanced techniques serve a dual purpose: they not only potentially enhance the model's accuracy and generalization capabilities but also demonstrate the immense potential of transformer-based models in tackling complex sentiment analysis tasks. By employing these methods, we've showcased how to harness the full power of state-of-the-art NLP architectures, providing a solid and extensible foundation for further exploration and refinement.

The knowledge and experience gained from this project open up numerous avenues for application in real-world NLP scenarios. From analyzing customer feedback and social media sentiment to gauging public opinion on various topics, the techniques explored here have far-reaching implications. This project serves as a springboard for data scientists and NLP practitioners to dive deeper into the fascinating world of sentiment analysis, encouraging further innovation and advancement in this critical area of artificial intelligence and machine learning.

9.2 Project 2: Sentiment Analysis Using Transformer-based Models

Sentiment analysis is a fundamental and highly significant task within the field of Natural Language Processing (NLP), focusing on the intricate process of deciphering and interpreting the underlying emotional tone or attitude expressed in a given piece of text.

This project delves into the application of cutting-edge transformer-based models, with a particular emphasis on BERT (Bidirectional Encoder Representations from Transformers), to conduct sophisticated sentiment analysis on textual data.

By leveraging these advanced neural network architectures, we aim to develop a robust system capable of accurately discerning and categorizing the sentiments conveyed in various forms of written communication, ranging from social media posts and product reviews to news articles and beyond.

9.2.1 Problem Statement and Dataset

For this project, we will utilize the IMDB Movie Reviews dataset, a comprehensive collection comprising 50,000 movie reviews. Each review in this dataset has been meticulously labeled as either positive or negative, providing a rich source of sentiment-annotated data.

Our primary objective is to develop and train a sophisticated model capable of accurately discerning and classifying the underlying sentiment expressed in these movie reviews. This task presents an excellent opportunity to apply advanced natural language processing techniques to real-world textual data, with the ultimate aim of creating a robust sentiment analysis system that can effectively interpret the nuanced opinions and emotions conveyed in written film critiques.

Loading and Exploring the Dataset

import pandas as pd
from datasets import load_dataset
from sklearn.model_selection import train_test_split

# Load the IMDB dataset
dataset = load_dataset('imdb')

# Convert to pandas DataFrame for easier manipulation
train_df = pd.DataFrame(dataset['train'])
test_df = pd.DataFrame(dataset['test'])

# Display basic information about the dataset
print(train_df.info())
print(train_df['label'].value_counts(normalize=True))

# Display a few examples
print(train_df.head())

Here's a breakdown of what it does:

  • Imports necessary libraries: pandas for data manipulation, load_dataset from the datasets library to load the IMDB dataset, and train_test_split from sklearn for data splitting (although not used in this snippet)
  • Loads the IMDB dataset using load_dataset('imdb')
  • Converts the train and test sets to pandas DataFrames for easier manipulation
  • Displays basic information about the training dataset using train_df.info()
  • Shows the distribution of labels (positive/negative sentiment) in the training set using train_df['label'].value_counts(normalize=True)
  • Displays the first few examples from the training set using train_df.head()

9.2.2 Data Preprocessing

Before we can feed our data into the BERT model for analysis, it is crucial to undergo a comprehensive preprocessing stage. This essential step involves several key processes that prepare the raw text data for optimal processing by our advanced neural network.

The primary components of this preprocessing phase include tokenization, which breaks down the text into individual units or tokens that the model can interpret; padding, which ensures all input sequences are of uniform length for batch processing; and the creation of attention masks, which guide the model's focus to relevant parts of the input while ignoring padding tokens. 

These preprocessing steps are fundamental in transforming our raw textual data into a format that can be efficiently and effectively processed by our BERT model, ultimately enabling more accurate sentiment analysis results.

from transformers import BertTokenizer
import torch

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

def preprocess_data(texts, labels, max_length=256):
    encoded = tokenizer.batch_encode_plus(
        texts,
        add_special_tokens=True,
        max_length=max_length,
        padding='max_length',
        truncation=True,
        return_attention_mask=True,
        return_tensors='pt'
    )
    return {
        'input_ids': encoded['input_ids'],
        'attention_mask': encoded['attention_mask'],
        'labels': torch.tensor(labels)
    }

# Preprocess the data
train_data = preprocess_data(train_df['text'].tolist(), train_df['label'].tolist())
test_data = preprocess_data(test_df['text'].tolist(), test_df['label'].tolist())

Here's a breakdown of what the code does:

  • It imports the necessary libraries: BertTokenizer from transformers and torch
  • It initializes a BERT tokenizer using the 'bert-base-uncased' model
  • The preprocess_data function is defined, which takes texts and labels as input, along with an optional max_length parameter
  • Inside the function, it uses the tokenizer's batch_encode_plus method to encode the input texts. This method:
    • Adds special tokens (like [CLS] and [SEP])
    • Pads or truncates sequences to a maximum length
    • Creates attention masks
    • Returns tensors suitable for PyTorch
  • The function returns a dictionary containing:
    • input_ids: the encoded and padded text sequences
    • attention_mask: a mask indicating which tokens are padding (0) and which are not (1)
    • labels: the sentiment labels converted to a PyTorch tensor
  • Finally, the code applies this preprocessing function to both the training and testing data, creating train_data and test_data

This preprocessing step is crucial as it transforms the raw text data into a format that can be efficiently processed by the BERT model for sentiment analysis

9.2.3 Building and Training the BERT Model

For this project, we will harness the power of the BertForSequenceClassification model, a sophisticated tool available in the transformers library. This advanced model is meticulously engineered and optimized for text classification tasks, making it an ideal choice for our sentiment analysis endeavor.

By leveraging this state-of-the-art architecture, we can effectively capture the nuanced sentiments expressed in our movie review dataset, enabling highly accurate classification of positive and negative sentiments.

from transformers import BertForSequenceClassification, AdamW
from torch.utils.data import DataLoader, TensorDataset
import torch.nn as nn

# Set up the model
model = BertForSequenceClassification.from_pretrained('bert-base-uncased', num_labels=2)

# Set up the optimizer
optimizer = AdamW(model.parameters(), lr=2e-5)

# Create DataLoader
train_dataset = TensorDataset(train_data['input_ids'], train_data['attention_mask'], train_data['labels'])
train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True)

# Training loop
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model.to(device)

num_epochs = 3
for epoch in range(num_epochs):
    model.train()
    for batch in train_loader:
        input_ids, attention_mask, labels = [b.to(device) for b in batch]
        
        optimizer.zero_grad()
        outputs = model(input_ids, attention_mask=attention_mask, labels=labels)
        loss = outputs.loss
        loss.backward()
        optimizer.step()
    
    print(f"Epoch {epoch+1}/{num_epochs} completed")

# Save the model
torch.save(model.state_dict(), 'bert_sentiment_model.pth')

Here's a breakdown of the main components:

  1. Model Setup: The code initializes a BertForSequenceClassification model, which is pre-trained and fine-tuned for sequence classification tasks like sentiment analysis.
  2. Optimizer: It sets up an AdamW optimizer, which is an improved version of Adam, commonly used for training deep learning models.
  3. Data Preparation: The code creates a TensorDataset and a DataLoader to efficiently batch and shuffle the training data.
  4. Training Loop: The model is trained for 3 epochs. In each epoch:
    • - It iterates through batches of data
    • - Computes the loss
    • - Performs backpropagation
    • - Updates the model parameters
  5. Device Utilization: The code checks for GPU availability and moves the model to the appropriate device (CPU or GPU) for efficient computation.
  6. Model Saving: After training, the model's state dictionary is saved to a file for future use.

This implementation allows for effective training of a BERT model on the IMDB movie review dataset, enabling it to learn and classify the sentiment (positive or negative) of movie reviews.

9.2.4 Evaluating the Model

After completing the training phase, it is crucial to assess our model's effectiveness and accuracy by evaluating its performance on the test set. This evaluation step allows us to gauge how well our BERT-based sentiment analysis model generalizes to unseen data and provides valuable insights into its real-world applicability.

By analyzing various metrics such as accuracy, precision, recall, and F1-score, we can gain a comprehensive understanding of our model's strengths and potential areas for improvement.

from sklearn.metrics import accuracy_score, classification_report

model.eval()
test_dataset = TensorDataset(test_data['input_ids'], test_data['attention_mask'], test_data['labels'])
test_loader = DataLoader(test_dataset, batch_size=32)

all_preds = []
all_labels = []

with torch.no_grad():
    for batch in test_loader:
        input_ids, attention_mask, labels = [b.to(device) for b in batch]
        outputs = model(input_ids, attention_mask=attention_mask)
        preds = torch.argmax(outputs.logits, dim=1)
        all_preds.extend(preds.cpu().numpy())
        all_labels.extend(labels.cpu().numpy())

accuracy = accuracy_score(all_labels, all_preds)
print(f"Accuracy: {accuracy}")
print(classification_report(all_labels, all_preds))

Here's a breakdown of what it does:

  • It imports necessary metrics from sklearn for model evaluation
  • Sets the model to evaluation mode with model.eval()
  • Creates a TensorDataset and DataLoader for the test data, which helps in batch processing
  • Initializes empty lists to store all predictions and true labels
  • Uses a with torch.no_grad() context to disable gradient calculations during inference, which saves memory and speeds up computation
  • Iterates through the test data in batches:
    • Moves the input data to the appropriate device (CPU or GPU)
    • Generates predictions using the model
    • Extracts the predicted class (sentiment) for each sample
    • Adds the predictions and true labels to their respective lists
  • Calculates the overall accuracy of the model using accuracy_score
  • Prints a detailed classification report, which typically includes precision, recall, and F1-score for each class

This evaluation process allows us to assess how well the model performs on unseen data, giving us insights into its effectiveness for sentiment analysis tasks.

9.2.5 Inference with New Text

With our model now fully trained and optimized, we can harness its capabilities to analyze and predict the sentiment of new, previously unseen text. This practical application of our sentiment analysis model allows us to gain valuable insights from fresh, real-world data, demonstrating the model's effectiveness beyond the training dataset.

By leveraging the power of our fine-tuned BERT-based model, we can now confidently assess the emotional tone of various text inputs, ranging from customer reviews and social media posts to news articles and beyond, providing a robust tool for understanding public opinion and consumer sentiment across diverse domains.

def predict_sentiment(text):
    encoded = tokenizer.encode_plus(
        text,
        add_special_tokens=True,
        max_length=256,
        padding='max_length',
        truncation=True,
        return_attention_mask=True,
        return_tensors='pt'
    )
    
    input_ids = encoded['input_ids'].to(device)
    attention_mask = encoded['attention_mask'].to(device)
    
    with torch.no_grad():
        outputs = model(input_ids, attention_mask=attention_mask)
        pred = torch.argmax(outputs.logits, dim=1)
    
    return "Positive" if pred.item() == 1 else "Negative"

# Example usage
new_review = "This movie was absolutely fantastic! I loved every minute of it."
sentiment = predict_sentiment(new_review)
print(f"Predicted sentiment: {sentiment}")

Here's a breakdown of what the code does:

  1. The function takes a text input and preprocesses it using the BERT tokenizer
  2. It encodes the text, adding special tokens, padding, and creating an attention mask
  3. The encoded input is then passed through the BERT model to get predictions
  4. The model's output logits are used to determine the sentiment (positive or negative)
  5. The function returns "Positive" if the prediction is 1, and "Negative" otherwise

The code also includes an example usage of the function:

  1. A sample review is provided: "This movie was absolutely fantastic! I loved every minute of it."
  2. The predict_sentiment function is called with this review
  3. The predicted sentiment is then printed

This function allows for easy sentiment analysis of new, unseen text using the trained BERT model, demonstrating its practical application for analyzing various text inputs such as customer reviews or social media posts

9.2.6 Advanced Techniques

Fine-tuning with Discriminative Learning Rates

To enhance our model's performance, we can implement discriminative learning rates, a sophisticated technique where different components of the model are trained at varying rates. This approach allows for more nuanced optimization, as it recognizes that different layers of the neural network may require different learning paces.

By applying higher learning rates to the upper layers of the model, which are more task-specific, and lower rates to the lower layers, which capture more general features, we can fine-tune the model more effectively.

This method is particularly beneficial when working with pre-trained models like BERT, as it allows us to carefully adjust the model's parameters without disrupting the valuable information learned during pre-training.

from transformers import get_linear_schedule_with_warmup

# Prepare optimizer and schedule (linear warmup and decay)
no_decay = ['bias', 'LayerNorm.weight']
optimizer_grouped_parameters = [
    {'params': [p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)], 'weight_decay': 0.01},
    {'params': [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)], 'weight_decay': 0.0}
]

optimizer = AdamW(optimizer_grouped_parameters, lr=2e-5, eps=1e-8)
scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps=0, num_training_steps=len(train_loader) * num_epochs)

# Update the training loop to use the scheduler
for epoch in range(num_epochs):
    model.train()
    for batch in train_loader:
        input_ids, attention_mask, labels = [b.to(device) for b in batch]
        
        optimizer.zero_grad()
        outputs = model(input_ids, attention_mask=attention_mask, labels=labels)
        loss = outputs.loss
        loss.backward()
        optimizer.step()
        scheduler.step()
    
    print(f"Epoch {epoch+1}/{num_epochs} completed")

Here's a breakdown of the key components:

  1. Importing the scheduler: The code imports get_linear_schedule_with_warmup from the transformers library.
  2. Optimizer configuration: 
    • - It sets up two groups of parameters: one with weight decay and one without.
    • - This approach helps in applying different learning rates to different parts of the model.
  3. Optimizer and scheduler initialization: 
    • - The AdamW optimizer is initialized with the grouped parameters.
    • - A linear learning rate scheduler with warmup is created, which will adjust the learning rate during training.
  4. Training loop: 
    • - The code updates the training loop to incorporate the scheduler.
    • - After each optimization step, the scheduler is stepped to adjust the learning rate.

This implementation allows for more effective fine-tuning of the BERT model by applying different learning rates to different parts of the model and gradually adjusting the learning rate throughout the training process.

Data Augmentation

To enhance our dataset and potentially improve model performance, we can employ various data augmentation techniques. One particularly effective method is back-translation, which involves translating the original text to another language and then back to the original language. This process introduces subtle variations in the text while preserving its overall meaning and sentiment.

Additionally, we can explore other augmentation strategies such as synonym replacement, random insertion or deletion of words, and text paraphrasing. These techniques collectively help to increase the diversity and size of our training data, potentially leading to a more robust and generalizable sentiment analysis model.

from transformers import MarianMTModel, MarianTokenizer

# Load translation models
en_to_fr = MarianMTModel.from_pretrained('Helsinki-NLP/opus-mt-en-fr')
fr_to_en = MarianMTModel.from_pretrained('Helsinki-NLP/opus-mt-fr-en')
en_tokenizer = MarianTokenizer.from_pretrained('Helsinki-NLP/opus-mt-en-fr')
fr_tokenizer = MarianTokenizer.from_pretrained('Helsinki-NLP/opus-mt-fr-en')

def back_translate(text):
    # Translate to French
    fr_text = en_to_fr.generate(**en_tokenizer(text, return_tensors="pt", padding=True))
    fr_text = [en_tokenizer.decode(t, skip_special_tokens=True) for t in fr_text][0]
    
    # Translate back to English
    en_text = fr_to_en.generate(**fr_tokenizer(fr_text, return_tensors="pt", padding=True))
    en_text = [fr_tokenizer.decode(t, skip_special_tokens=True) for t in en_text][0]
    
    return en_text

# Augment the training data
augmented_texts = [back_translate(text) for text in train_df['text'][:1000]]  # Augment first 1000 samples
augmented_labels = train_df['label'][:1000]

train_df = pd.concat([train_df, pd.DataFrame({'text': augmented_texts, 'label': augmented_labels})])

Here's an explanation of the key components:

  • The code imports necessary models and tokenizers from the Transformers library for translation tasks.
  • It loads pre-trained models for English-to-French and French-to-English translation.
  • The back_translate function is defined to perform the augmentation:
    • It translates the input English text to French
    • Then translates the French text back to English
    • This process introduces subtle variations while preserving the overall meaning
  • The code then augments the training data:
    • It applies back-translation to the first 1000 samples of the training data
    • The augmented texts and their corresponding labels are added to the training dataset

This technique helps increase the diversity of the training data, potentially leading to a more robust and generalizable sentiment analysis model.

Ensemble Methods

To potentially enhance our model's performance and robustness, we can implement an ensemble approach. This technique involves creating multiple models, each with its own strengths and characteristics, and combining their predictions to generate a more accurate and reliable final output.

By leveraging the collective intelligence of various models, we can often achieve better results than relying on a single model alone. This ensemble method can help mitigate individual model weaknesses and capture a broader range of patterns in the data, ultimately leading to improved sentiment analysis accuracy.

# Train multiple models (e.g., BERT, RoBERTa, DistilBERT)
from transformers import RobertaForSequenceClassification, DistilBertForSequenceClassification

models = [
    BertForSequenceClassification.from_pretrained('bert-base-uncased', num_labels=2),
    RobertaForSequenceClassification.from_pretrained('roberta-base', num_labels=2),
    DistilBertForSequenceClassification.from_pretrained('distilbert-base-uncased', num_labels=2)
]

# Train each model (code omitted for brevity)

def ensemble_predict(text):
    encoded = tokenizer.encode_plus(
        text,
        add_special_tokens=True,
        max_length=256,
        padding='max_length',
        truncation=True,
        return_attention_mask=True,
        return_tensors='pt'
    )
    
    input_ids = encoded['input_ids'].to(device)
    attention_mask = encoded['attention_mask'].to(device)
    
    predictions = []
    with torch.no_grad():
        for model in models:
            outputs = model(input_ids, attention_mask=attention_mask)
            pred = torch.softmax(outputs.logits, dim=1)
            predictions.append(pred)
    
    # Average predictions
    avg_pred = torch.mean(torch.stack(predictions), dim=0)
    final_pred = torch.argmax(avg_pred, dim=1)
    
    return "Positive" if final_pred.item() == 1 else "Negative"

This code implements an ensemble method for sentiment analysis using multiple transformer-based models. Here's a breakdown of its key components:

  1. Model Initialization: The code imports and initializes three different pre-trained models: BERT, RoBERTa, and DistilBERT. Each model is set up for binary classification (positive/negative sentiment).
  2. Ensemble Prediction Function: The ensemble_predict function is defined to make predictions using all three models:
    • It tokenizes and encodes the input text using a tokenizer (presumably BERT's tokenizer, though this isn't explicitly shown in the snippet).
    • The encoded input is then passed through each model to get predictions.
    • The raw logits from each model are converted to probabilities using softmax.
    • The predictions from all models are averaged to get a final prediction.
    • The function returns "Positive" or "Negative" based on the averaged prediction.

This ensemble approach aims to improve prediction accuracy by combining the strengths of multiple models, potentially leading to more robust sentiment analysis results.

9.2.7 Conclusion

In this enhanced project, we have successfully implemented a sophisticated sentiment analysis model leveraging the power of BERT architecture. We've delved into advanced techniques to significantly boost its performance and versatility. Our comprehensive approach covered crucial aspects including meticulous data preprocessing, rigorous model training, thorough evaluation, and seamless inference processes.

Furthermore, we've introduced and explored cutting-edge techniques to push the boundaries of model performance. These include the implementation of discriminative learning rates, which allow for fine-tuned optimization across different layers of the model. We've also incorporated data augmentation strategies, particularly back-translation, to enrich our dataset and improve the model's ability to generalize. Additionally, we've ventured into ensemble methods, combining the strengths of multiple models to achieve more robust and accurate predictions.

These advanced techniques serve a dual purpose: they not only potentially enhance the model's accuracy and generalization capabilities but also demonstrate the immense potential of transformer-based models in tackling complex sentiment analysis tasks. By employing these methods, we've showcased how to harness the full power of state-of-the-art NLP architectures, providing a solid and extensible foundation for further exploration and refinement.

The knowledge and experience gained from this project open up numerous avenues for application in real-world NLP scenarios. From analyzing customer feedback and social media sentiment to gauging public opinion on various topics, the techniques explored here have far-reaching implications. This project serves as a springboard for data scientists and NLP practitioners to dive deeper into the fascinating world of sentiment analysis, encouraging further innovation and advancement in this critical area of artificial intelligence and machine learning.

9.2 Project 2: Sentiment Analysis Using Transformer-based Models

Sentiment analysis is a fundamental and highly significant task within the field of Natural Language Processing (NLP), focusing on the intricate process of deciphering and interpreting the underlying emotional tone or attitude expressed in a given piece of text.

This project delves into the application of cutting-edge transformer-based models, with a particular emphasis on BERT (Bidirectional Encoder Representations from Transformers), to conduct sophisticated sentiment analysis on textual data.

By leveraging these advanced neural network architectures, we aim to develop a robust system capable of accurately discerning and categorizing the sentiments conveyed in various forms of written communication, ranging from social media posts and product reviews to news articles and beyond.

9.2.1 Problem Statement and Dataset

For this project, we will utilize the IMDB Movie Reviews dataset, a comprehensive collection comprising 50,000 movie reviews. Each review in this dataset has been meticulously labeled as either positive or negative, providing a rich source of sentiment-annotated data.

Our primary objective is to develop and train a sophisticated model capable of accurately discerning and classifying the underlying sentiment expressed in these movie reviews. This task presents an excellent opportunity to apply advanced natural language processing techniques to real-world textual data, with the ultimate aim of creating a robust sentiment analysis system that can effectively interpret the nuanced opinions and emotions conveyed in written film critiques.

Loading and Exploring the Dataset

import pandas as pd
from datasets import load_dataset
from sklearn.model_selection import train_test_split

# Load the IMDB dataset
dataset = load_dataset('imdb')

# Convert to pandas DataFrame for easier manipulation
train_df = pd.DataFrame(dataset['train'])
test_df = pd.DataFrame(dataset['test'])

# Display basic information about the dataset
print(train_df.info())
print(train_df['label'].value_counts(normalize=True))

# Display a few examples
print(train_df.head())

Here's a breakdown of what it does:

  • Imports necessary libraries: pandas for data manipulation, load_dataset from the datasets library to load the IMDB dataset, and train_test_split from sklearn for data splitting (although not used in this snippet)
  • Loads the IMDB dataset using load_dataset('imdb')
  • Converts the train and test sets to pandas DataFrames for easier manipulation
  • Displays basic information about the training dataset using train_df.info()
  • Shows the distribution of labels (positive/negative sentiment) in the training set using train_df['label'].value_counts(normalize=True)
  • Displays the first few examples from the training set using train_df.head()

9.2.2 Data Preprocessing

Before we can feed our data into the BERT model for analysis, it is crucial to undergo a comprehensive preprocessing stage. This essential step involves several key processes that prepare the raw text data for optimal processing by our advanced neural network.

The primary components of this preprocessing phase include tokenization, which breaks down the text into individual units or tokens that the model can interpret; padding, which ensures all input sequences are of uniform length for batch processing; and the creation of attention masks, which guide the model's focus to relevant parts of the input while ignoring padding tokens. 

These preprocessing steps are fundamental in transforming our raw textual data into a format that can be efficiently and effectively processed by our BERT model, ultimately enabling more accurate sentiment analysis results.

from transformers import BertTokenizer
import torch

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

def preprocess_data(texts, labels, max_length=256):
    encoded = tokenizer.batch_encode_plus(
        texts,
        add_special_tokens=True,
        max_length=max_length,
        padding='max_length',
        truncation=True,
        return_attention_mask=True,
        return_tensors='pt'
    )
    return {
        'input_ids': encoded['input_ids'],
        'attention_mask': encoded['attention_mask'],
        'labels': torch.tensor(labels)
    }

# Preprocess the data
train_data = preprocess_data(train_df['text'].tolist(), train_df['label'].tolist())
test_data = preprocess_data(test_df['text'].tolist(), test_df['label'].tolist())

Here's a breakdown of what the code does:

  • It imports the necessary libraries: BertTokenizer from transformers and torch
  • It initializes a BERT tokenizer using the 'bert-base-uncased' model
  • The preprocess_data function is defined, which takes texts and labels as input, along with an optional max_length parameter
  • Inside the function, it uses the tokenizer's batch_encode_plus method to encode the input texts. This method:
    • Adds special tokens (like [CLS] and [SEP])
    • Pads or truncates sequences to a maximum length
    • Creates attention masks
    • Returns tensors suitable for PyTorch
  • The function returns a dictionary containing:
    • input_ids: the encoded and padded text sequences
    • attention_mask: a mask indicating which tokens are padding (0) and which are not (1)
    • labels: the sentiment labels converted to a PyTorch tensor
  • Finally, the code applies this preprocessing function to both the training and testing data, creating train_data and test_data

This preprocessing step is crucial as it transforms the raw text data into a format that can be efficiently processed by the BERT model for sentiment analysis

9.2.3 Building and Training the BERT Model

For this project, we will harness the power of the BertForSequenceClassification model, a sophisticated tool available in the transformers library. This advanced model is meticulously engineered and optimized for text classification tasks, making it an ideal choice for our sentiment analysis endeavor.

By leveraging this state-of-the-art architecture, we can effectively capture the nuanced sentiments expressed in our movie review dataset, enabling highly accurate classification of positive and negative sentiments.

from transformers import BertForSequenceClassification, AdamW
from torch.utils.data import DataLoader, TensorDataset
import torch.nn as nn

# Set up the model
model = BertForSequenceClassification.from_pretrained('bert-base-uncased', num_labels=2)

# Set up the optimizer
optimizer = AdamW(model.parameters(), lr=2e-5)

# Create DataLoader
train_dataset = TensorDataset(train_data['input_ids'], train_data['attention_mask'], train_data['labels'])
train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True)

# Training loop
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model.to(device)

num_epochs = 3
for epoch in range(num_epochs):
    model.train()
    for batch in train_loader:
        input_ids, attention_mask, labels = [b.to(device) for b in batch]
        
        optimizer.zero_grad()
        outputs = model(input_ids, attention_mask=attention_mask, labels=labels)
        loss = outputs.loss
        loss.backward()
        optimizer.step()
    
    print(f"Epoch {epoch+1}/{num_epochs} completed")

# Save the model
torch.save(model.state_dict(), 'bert_sentiment_model.pth')

Here's a breakdown of the main components:

  1. Model Setup: The code initializes a BertForSequenceClassification model, which is pre-trained and fine-tuned for sequence classification tasks like sentiment analysis.
  2. Optimizer: It sets up an AdamW optimizer, which is an improved version of Adam, commonly used for training deep learning models.
  3. Data Preparation: The code creates a TensorDataset and a DataLoader to efficiently batch and shuffle the training data.
  4. Training Loop: The model is trained for 3 epochs. In each epoch:
    • - It iterates through batches of data
    • - Computes the loss
    • - Performs backpropagation
    • - Updates the model parameters
  5. Device Utilization: The code checks for GPU availability and moves the model to the appropriate device (CPU or GPU) for efficient computation.
  6. Model Saving: After training, the model's state dictionary is saved to a file for future use.

This implementation allows for effective training of a BERT model on the IMDB movie review dataset, enabling it to learn and classify the sentiment (positive or negative) of movie reviews.

9.2.4 Evaluating the Model

After completing the training phase, it is crucial to assess our model's effectiveness and accuracy by evaluating its performance on the test set. This evaluation step allows us to gauge how well our BERT-based sentiment analysis model generalizes to unseen data and provides valuable insights into its real-world applicability.

By analyzing various metrics such as accuracy, precision, recall, and F1-score, we can gain a comprehensive understanding of our model's strengths and potential areas for improvement.

from sklearn.metrics import accuracy_score, classification_report

model.eval()
test_dataset = TensorDataset(test_data['input_ids'], test_data['attention_mask'], test_data['labels'])
test_loader = DataLoader(test_dataset, batch_size=32)

all_preds = []
all_labels = []

with torch.no_grad():
    for batch in test_loader:
        input_ids, attention_mask, labels = [b.to(device) for b in batch]
        outputs = model(input_ids, attention_mask=attention_mask)
        preds = torch.argmax(outputs.logits, dim=1)
        all_preds.extend(preds.cpu().numpy())
        all_labels.extend(labels.cpu().numpy())

accuracy = accuracy_score(all_labels, all_preds)
print(f"Accuracy: {accuracy}")
print(classification_report(all_labels, all_preds))

Here's a breakdown of what it does:

  • It imports necessary metrics from sklearn for model evaluation
  • Sets the model to evaluation mode with model.eval()
  • Creates a TensorDataset and DataLoader for the test data, which helps in batch processing
  • Initializes empty lists to store all predictions and true labels
  • Uses a with torch.no_grad() context to disable gradient calculations during inference, which saves memory and speeds up computation
  • Iterates through the test data in batches:
    • Moves the input data to the appropriate device (CPU or GPU)
    • Generates predictions using the model
    • Extracts the predicted class (sentiment) for each sample
    • Adds the predictions and true labels to their respective lists
  • Calculates the overall accuracy of the model using accuracy_score
  • Prints a detailed classification report, which typically includes precision, recall, and F1-score for each class

This evaluation process allows us to assess how well the model performs on unseen data, giving us insights into its effectiveness for sentiment analysis tasks.

9.2.5 Inference with New Text

With our model now fully trained and optimized, we can harness its capabilities to analyze and predict the sentiment of new, previously unseen text. This practical application of our sentiment analysis model allows us to gain valuable insights from fresh, real-world data, demonstrating the model's effectiveness beyond the training dataset.

By leveraging the power of our fine-tuned BERT-based model, we can now confidently assess the emotional tone of various text inputs, ranging from customer reviews and social media posts to news articles and beyond, providing a robust tool for understanding public opinion and consumer sentiment across diverse domains.

def predict_sentiment(text):
    encoded = tokenizer.encode_plus(
        text,
        add_special_tokens=True,
        max_length=256,
        padding='max_length',
        truncation=True,
        return_attention_mask=True,
        return_tensors='pt'
    )
    
    input_ids = encoded['input_ids'].to(device)
    attention_mask = encoded['attention_mask'].to(device)
    
    with torch.no_grad():
        outputs = model(input_ids, attention_mask=attention_mask)
        pred = torch.argmax(outputs.logits, dim=1)
    
    return "Positive" if pred.item() == 1 else "Negative"

# Example usage
new_review = "This movie was absolutely fantastic! I loved every minute of it."
sentiment = predict_sentiment(new_review)
print(f"Predicted sentiment: {sentiment}")

Here's a breakdown of what the code does:

  1. The function takes a text input and preprocesses it using the BERT tokenizer
  2. It encodes the text, adding special tokens, padding, and creating an attention mask
  3. The encoded input is then passed through the BERT model to get predictions
  4. The model's output logits are used to determine the sentiment (positive or negative)
  5. The function returns "Positive" if the prediction is 1, and "Negative" otherwise

The code also includes an example usage of the function:

  1. A sample review is provided: "This movie was absolutely fantastic! I loved every minute of it."
  2. The predict_sentiment function is called with this review
  3. The predicted sentiment is then printed

This function allows for easy sentiment analysis of new, unseen text using the trained BERT model, demonstrating its practical application for analyzing various text inputs such as customer reviews or social media posts

9.2.6 Advanced Techniques

Fine-tuning with Discriminative Learning Rates

To enhance our model's performance, we can implement discriminative learning rates, a sophisticated technique where different components of the model are trained at varying rates. This approach allows for more nuanced optimization, as it recognizes that different layers of the neural network may require different learning paces.

By applying higher learning rates to the upper layers of the model, which are more task-specific, and lower rates to the lower layers, which capture more general features, we can fine-tune the model more effectively.

This method is particularly beneficial when working with pre-trained models like BERT, as it allows us to carefully adjust the model's parameters without disrupting the valuable information learned during pre-training.

from transformers import get_linear_schedule_with_warmup

# Prepare optimizer and schedule (linear warmup and decay)
no_decay = ['bias', 'LayerNorm.weight']
optimizer_grouped_parameters = [
    {'params': [p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)], 'weight_decay': 0.01},
    {'params': [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)], 'weight_decay': 0.0}
]

optimizer = AdamW(optimizer_grouped_parameters, lr=2e-5, eps=1e-8)
scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps=0, num_training_steps=len(train_loader) * num_epochs)

# Update the training loop to use the scheduler
for epoch in range(num_epochs):
    model.train()
    for batch in train_loader:
        input_ids, attention_mask, labels = [b.to(device) for b in batch]
        
        optimizer.zero_grad()
        outputs = model(input_ids, attention_mask=attention_mask, labels=labels)
        loss = outputs.loss
        loss.backward()
        optimizer.step()
        scheduler.step()
    
    print(f"Epoch {epoch+1}/{num_epochs} completed")

Here's a breakdown of the key components:

  1. Importing the scheduler: The code imports get_linear_schedule_with_warmup from the transformers library.
  2. Optimizer configuration: 
    • - It sets up two groups of parameters: one with weight decay and one without.
    • - This approach helps in applying different learning rates to different parts of the model.
  3. Optimizer and scheduler initialization: 
    • - The AdamW optimizer is initialized with the grouped parameters.
    • - A linear learning rate scheduler with warmup is created, which will adjust the learning rate during training.
  4. Training loop: 
    • - The code updates the training loop to incorporate the scheduler.
    • - After each optimization step, the scheduler is stepped to adjust the learning rate.

This implementation allows for more effective fine-tuning of the BERT model by applying different learning rates to different parts of the model and gradually adjusting the learning rate throughout the training process.

Data Augmentation

To enhance our dataset and potentially improve model performance, we can employ various data augmentation techniques. One particularly effective method is back-translation, which involves translating the original text to another language and then back to the original language. This process introduces subtle variations in the text while preserving its overall meaning and sentiment.

Additionally, we can explore other augmentation strategies such as synonym replacement, random insertion or deletion of words, and text paraphrasing. These techniques collectively help to increase the diversity and size of our training data, potentially leading to a more robust and generalizable sentiment analysis model.

from transformers import MarianMTModel, MarianTokenizer

# Load translation models
en_to_fr = MarianMTModel.from_pretrained('Helsinki-NLP/opus-mt-en-fr')
fr_to_en = MarianMTModel.from_pretrained('Helsinki-NLP/opus-mt-fr-en')
en_tokenizer = MarianTokenizer.from_pretrained('Helsinki-NLP/opus-mt-en-fr')
fr_tokenizer = MarianTokenizer.from_pretrained('Helsinki-NLP/opus-mt-fr-en')

def back_translate(text):
    # Translate to French
    fr_text = en_to_fr.generate(**en_tokenizer(text, return_tensors="pt", padding=True))
    fr_text = [en_tokenizer.decode(t, skip_special_tokens=True) for t in fr_text][0]
    
    # Translate back to English
    en_text = fr_to_en.generate(**fr_tokenizer(fr_text, return_tensors="pt", padding=True))
    en_text = [fr_tokenizer.decode(t, skip_special_tokens=True) for t in en_text][0]
    
    return en_text

# Augment the training data
augmented_texts = [back_translate(text) for text in train_df['text'][:1000]]  # Augment first 1000 samples
augmented_labels = train_df['label'][:1000]

train_df = pd.concat([train_df, pd.DataFrame({'text': augmented_texts, 'label': augmented_labels})])

Here's an explanation of the key components:

  • The code imports necessary models and tokenizers from the Transformers library for translation tasks.
  • It loads pre-trained models for English-to-French and French-to-English translation.
  • The back_translate function is defined to perform the augmentation:
    • It translates the input English text to French
    • Then translates the French text back to English
    • This process introduces subtle variations while preserving the overall meaning
  • The code then augments the training data:
    • It applies back-translation to the first 1000 samples of the training data
    • The augmented texts and their corresponding labels are added to the training dataset

This technique helps increase the diversity of the training data, potentially leading to a more robust and generalizable sentiment analysis model.

Ensemble Methods

To potentially enhance our model's performance and robustness, we can implement an ensemble approach. This technique involves creating multiple models, each with its own strengths and characteristics, and combining their predictions to generate a more accurate and reliable final output.

By leveraging the collective intelligence of various models, we can often achieve better results than relying on a single model alone. This ensemble method can help mitigate individual model weaknesses and capture a broader range of patterns in the data, ultimately leading to improved sentiment analysis accuracy.

# Train multiple models (e.g., BERT, RoBERTa, DistilBERT)
from transformers import RobertaForSequenceClassification, DistilBertForSequenceClassification

models = [
    BertForSequenceClassification.from_pretrained('bert-base-uncased', num_labels=2),
    RobertaForSequenceClassification.from_pretrained('roberta-base', num_labels=2),
    DistilBertForSequenceClassification.from_pretrained('distilbert-base-uncased', num_labels=2)
]

# Train each model (code omitted for brevity)

def ensemble_predict(text):
    encoded = tokenizer.encode_plus(
        text,
        add_special_tokens=True,
        max_length=256,
        padding='max_length',
        truncation=True,
        return_attention_mask=True,
        return_tensors='pt'
    )
    
    input_ids = encoded['input_ids'].to(device)
    attention_mask = encoded['attention_mask'].to(device)
    
    predictions = []
    with torch.no_grad():
        for model in models:
            outputs = model(input_ids, attention_mask=attention_mask)
            pred = torch.softmax(outputs.logits, dim=1)
            predictions.append(pred)
    
    # Average predictions
    avg_pred = torch.mean(torch.stack(predictions), dim=0)
    final_pred = torch.argmax(avg_pred, dim=1)
    
    return "Positive" if final_pred.item() == 1 else "Negative"

This code implements an ensemble method for sentiment analysis using multiple transformer-based models. Here's a breakdown of its key components:

  1. Model Initialization: The code imports and initializes three different pre-trained models: BERT, RoBERTa, and DistilBERT. Each model is set up for binary classification (positive/negative sentiment).
  2. Ensemble Prediction Function: The ensemble_predict function is defined to make predictions using all three models:
    • It tokenizes and encodes the input text using a tokenizer (presumably BERT's tokenizer, though this isn't explicitly shown in the snippet).
    • The encoded input is then passed through each model to get predictions.
    • The raw logits from each model are converted to probabilities using softmax.
    • The predictions from all models are averaged to get a final prediction.
    • The function returns "Positive" or "Negative" based on the averaged prediction.

This ensemble approach aims to improve prediction accuracy by combining the strengths of multiple models, potentially leading to more robust sentiment analysis results.

9.2.7 Conclusion

In this enhanced project, we have successfully implemented a sophisticated sentiment analysis model leveraging the power of BERT architecture. We've delved into advanced techniques to significantly boost its performance and versatility. Our comprehensive approach covered crucial aspects including meticulous data preprocessing, rigorous model training, thorough evaluation, and seamless inference processes.

Furthermore, we've introduced and explored cutting-edge techniques to push the boundaries of model performance. These include the implementation of discriminative learning rates, which allow for fine-tuned optimization across different layers of the model. We've also incorporated data augmentation strategies, particularly back-translation, to enrich our dataset and improve the model's ability to generalize. Additionally, we've ventured into ensemble methods, combining the strengths of multiple models to achieve more robust and accurate predictions.

These advanced techniques serve a dual purpose: they not only potentially enhance the model's accuracy and generalization capabilities but also demonstrate the immense potential of transformer-based models in tackling complex sentiment analysis tasks. By employing these methods, we've showcased how to harness the full power of state-of-the-art NLP architectures, providing a solid and extensible foundation for further exploration and refinement.

The knowledge and experience gained from this project open up numerous avenues for application in real-world NLP scenarios. From analyzing customer feedback and social media sentiment to gauging public opinion on various topics, the techniques explored here have far-reaching implications. This project serves as a springboard for data scientists and NLP practitioners to dive deeper into the fascinating world of sentiment analysis, encouraging further innovation and advancement in this critical area of artificial intelligence and machine learning.

9.2 Project 2: Sentiment Analysis Using Transformer-based Models

Sentiment analysis is a fundamental and highly significant task within the field of Natural Language Processing (NLP), focusing on the intricate process of deciphering and interpreting the underlying emotional tone or attitude expressed in a given piece of text.

This project delves into the application of cutting-edge transformer-based models, with a particular emphasis on BERT (Bidirectional Encoder Representations from Transformers), to conduct sophisticated sentiment analysis on textual data.

By leveraging these advanced neural network architectures, we aim to develop a robust system capable of accurately discerning and categorizing the sentiments conveyed in various forms of written communication, ranging from social media posts and product reviews to news articles and beyond.

9.2.1 Problem Statement and Dataset

For this project, we will utilize the IMDB Movie Reviews dataset, a comprehensive collection comprising 50,000 movie reviews. Each review in this dataset has been meticulously labeled as either positive or negative, providing a rich source of sentiment-annotated data.

Our primary objective is to develop and train a sophisticated model capable of accurately discerning and classifying the underlying sentiment expressed in these movie reviews. This task presents an excellent opportunity to apply advanced natural language processing techniques to real-world textual data, with the ultimate aim of creating a robust sentiment analysis system that can effectively interpret the nuanced opinions and emotions conveyed in written film critiques.

Loading and Exploring the Dataset

import pandas as pd
from datasets import load_dataset
from sklearn.model_selection import train_test_split

# Load the IMDB dataset
dataset = load_dataset('imdb')

# Convert to pandas DataFrame for easier manipulation
train_df = pd.DataFrame(dataset['train'])
test_df = pd.DataFrame(dataset['test'])

# Display basic information about the dataset
print(train_df.info())
print(train_df['label'].value_counts(normalize=True))

# Display a few examples
print(train_df.head())

Here's a breakdown of what it does:

  • Imports necessary libraries: pandas for data manipulation, load_dataset from the datasets library to load the IMDB dataset, and train_test_split from sklearn for data splitting (although not used in this snippet)
  • Loads the IMDB dataset using load_dataset('imdb')
  • Converts the train and test sets to pandas DataFrames for easier manipulation
  • Displays basic information about the training dataset using train_df.info()
  • Shows the distribution of labels (positive/negative sentiment) in the training set using train_df['label'].value_counts(normalize=True)
  • Displays the first few examples from the training set using train_df.head()

9.2.2 Data Preprocessing

Before we can feed our data into the BERT model for analysis, it is crucial to undergo a comprehensive preprocessing stage. This essential step involves several key processes that prepare the raw text data for optimal processing by our advanced neural network.

The primary components of this preprocessing phase include tokenization, which breaks down the text into individual units or tokens that the model can interpret; padding, which ensures all input sequences are of uniform length for batch processing; and the creation of attention masks, which guide the model's focus to relevant parts of the input while ignoring padding tokens. 

These preprocessing steps are fundamental in transforming our raw textual data into a format that can be efficiently and effectively processed by our BERT model, ultimately enabling more accurate sentiment analysis results.

from transformers import BertTokenizer
import torch

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

def preprocess_data(texts, labels, max_length=256):
    encoded = tokenizer.batch_encode_plus(
        texts,
        add_special_tokens=True,
        max_length=max_length,
        padding='max_length',
        truncation=True,
        return_attention_mask=True,
        return_tensors='pt'
    )
    return {
        'input_ids': encoded['input_ids'],
        'attention_mask': encoded['attention_mask'],
        'labels': torch.tensor(labels)
    }

# Preprocess the data
train_data = preprocess_data(train_df['text'].tolist(), train_df['label'].tolist())
test_data = preprocess_data(test_df['text'].tolist(), test_df['label'].tolist())

Here's a breakdown of what the code does:

  • It imports the necessary libraries: BertTokenizer from transformers and torch
  • It initializes a BERT tokenizer using the 'bert-base-uncased' model
  • The preprocess_data function is defined, which takes texts and labels as input, along with an optional max_length parameter
  • Inside the function, it uses the tokenizer's batch_encode_plus method to encode the input texts. This method:
    • Adds special tokens (like [CLS] and [SEP])
    • Pads or truncates sequences to a maximum length
    • Creates attention masks
    • Returns tensors suitable for PyTorch
  • The function returns a dictionary containing:
    • input_ids: the encoded and padded text sequences
    • attention_mask: a mask indicating which tokens are padding (0) and which are not (1)
    • labels: the sentiment labels converted to a PyTorch tensor
  • Finally, the code applies this preprocessing function to both the training and testing data, creating train_data and test_data

This preprocessing step is crucial as it transforms the raw text data into a format that can be efficiently processed by the BERT model for sentiment analysis

9.2.3 Building and Training the BERT Model

For this project, we will harness the power of the BertForSequenceClassification model, a sophisticated tool available in the transformers library. This advanced model is meticulously engineered and optimized for text classification tasks, making it an ideal choice for our sentiment analysis endeavor.

By leveraging this state-of-the-art architecture, we can effectively capture the nuanced sentiments expressed in our movie review dataset, enabling highly accurate classification of positive and negative sentiments.

from transformers import BertForSequenceClassification, AdamW
from torch.utils.data import DataLoader, TensorDataset
import torch.nn as nn

# Set up the model
model = BertForSequenceClassification.from_pretrained('bert-base-uncased', num_labels=2)

# Set up the optimizer
optimizer = AdamW(model.parameters(), lr=2e-5)

# Create DataLoader
train_dataset = TensorDataset(train_data['input_ids'], train_data['attention_mask'], train_data['labels'])
train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True)

# Training loop
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model.to(device)

num_epochs = 3
for epoch in range(num_epochs):
    model.train()
    for batch in train_loader:
        input_ids, attention_mask, labels = [b.to(device) for b in batch]
        
        optimizer.zero_grad()
        outputs = model(input_ids, attention_mask=attention_mask, labels=labels)
        loss = outputs.loss
        loss.backward()
        optimizer.step()
    
    print(f"Epoch {epoch+1}/{num_epochs} completed")

# Save the model
torch.save(model.state_dict(), 'bert_sentiment_model.pth')

Here's a breakdown of the main components:

  1. Model Setup: The code initializes a BertForSequenceClassification model, which is pre-trained and fine-tuned for sequence classification tasks like sentiment analysis.
  2. Optimizer: It sets up an AdamW optimizer, which is an improved version of Adam, commonly used for training deep learning models.
  3. Data Preparation: The code creates a TensorDataset and a DataLoader to efficiently batch and shuffle the training data.
  4. Training Loop: The model is trained for 3 epochs. In each epoch:
    • - It iterates through batches of data
    • - Computes the loss
    • - Performs backpropagation
    • - Updates the model parameters
  5. Device Utilization: The code checks for GPU availability and moves the model to the appropriate device (CPU or GPU) for efficient computation.
  6. Model Saving: After training, the model's state dictionary is saved to a file for future use.

This implementation allows for effective training of a BERT model on the IMDB movie review dataset, enabling it to learn and classify the sentiment (positive or negative) of movie reviews.

9.2.4 Evaluating the Model

After completing the training phase, it is crucial to assess our model's effectiveness and accuracy by evaluating its performance on the test set. This evaluation step allows us to gauge how well our BERT-based sentiment analysis model generalizes to unseen data and provides valuable insights into its real-world applicability.

By analyzing various metrics such as accuracy, precision, recall, and F1-score, we can gain a comprehensive understanding of our model's strengths and potential areas for improvement.

from sklearn.metrics import accuracy_score, classification_report

model.eval()
test_dataset = TensorDataset(test_data['input_ids'], test_data['attention_mask'], test_data['labels'])
test_loader = DataLoader(test_dataset, batch_size=32)

all_preds = []
all_labels = []

with torch.no_grad():
    for batch in test_loader:
        input_ids, attention_mask, labels = [b.to(device) for b in batch]
        outputs = model(input_ids, attention_mask=attention_mask)
        preds = torch.argmax(outputs.logits, dim=1)
        all_preds.extend(preds.cpu().numpy())
        all_labels.extend(labels.cpu().numpy())

accuracy = accuracy_score(all_labels, all_preds)
print(f"Accuracy: {accuracy}")
print(classification_report(all_labels, all_preds))

Here's a breakdown of what it does:

  • It imports necessary metrics from sklearn for model evaluation
  • Sets the model to evaluation mode with model.eval()
  • Creates a TensorDataset and DataLoader for the test data, which helps in batch processing
  • Initializes empty lists to store all predictions and true labels
  • Uses a with torch.no_grad() context to disable gradient calculations during inference, which saves memory and speeds up computation
  • Iterates through the test data in batches:
    • Moves the input data to the appropriate device (CPU or GPU)
    • Generates predictions using the model
    • Extracts the predicted class (sentiment) for each sample
    • Adds the predictions and true labels to their respective lists
  • Calculates the overall accuracy of the model using accuracy_score
  • Prints a detailed classification report, which typically includes precision, recall, and F1-score for each class

This evaluation process allows us to assess how well the model performs on unseen data, giving us insights into its effectiveness for sentiment analysis tasks.

9.2.5 Inference with New Text

With our model now fully trained and optimized, we can harness its capabilities to analyze and predict the sentiment of new, previously unseen text. This practical application of our sentiment analysis model allows us to gain valuable insights from fresh, real-world data, demonstrating the model's effectiveness beyond the training dataset.

By leveraging the power of our fine-tuned BERT-based model, we can now confidently assess the emotional tone of various text inputs, ranging from customer reviews and social media posts to news articles and beyond, providing a robust tool for understanding public opinion and consumer sentiment across diverse domains.

def predict_sentiment(text):
    encoded = tokenizer.encode_plus(
        text,
        add_special_tokens=True,
        max_length=256,
        padding='max_length',
        truncation=True,
        return_attention_mask=True,
        return_tensors='pt'
    )
    
    input_ids = encoded['input_ids'].to(device)
    attention_mask = encoded['attention_mask'].to(device)
    
    with torch.no_grad():
        outputs = model(input_ids, attention_mask=attention_mask)
        pred = torch.argmax(outputs.logits, dim=1)
    
    return "Positive" if pred.item() == 1 else "Negative"

# Example usage
new_review = "This movie was absolutely fantastic! I loved every minute of it."
sentiment = predict_sentiment(new_review)
print(f"Predicted sentiment: {sentiment}")

Here's a breakdown of what the code does:

  1. The function takes a text input and preprocesses it using the BERT tokenizer
  2. It encodes the text, adding special tokens, padding, and creating an attention mask
  3. The encoded input is then passed through the BERT model to get predictions
  4. The model's output logits are used to determine the sentiment (positive or negative)
  5. The function returns "Positive" if the prediction is 1, and "Negative" otherwise

The code also includes an example usage of the function:

  1. A sample review is provided: "This movie was absolutely fantastic! I loved every minute of it."
  2. The predict_sentiment function is called with this review
  3. The predicted sentiment is then printed

This function allows for easy sentiment analysis of new, unseen text using the trained BERT model, demonstrating its practical application for analyzing various text inputs such as customer reviews or social media posts

9.2.6 Advanced Techniques

Fine-tuning with Discriminative Learning Rates

To enhance our model's performance, we can implement discriminative learning rates, a sophisticated technique where different components of the model are trained at varying rates. This approach allows for more nuanced optimization, as it recognizes that different layers of the neural network may require different learning paces.

By applying higher learning rates to the upper layers of the model, which are more task-specific, and lower rates to the lower layers, which capture more general features, we can fine-tune the model more effectively.

This method is particularly beneficial when working with pre-trained models like BERT, as it allows us to carefully adjust the model's parameters without disrupting the valuable information learned during pre-training.

from transformers import get_linear_schedule_with_warmup

# Prepare optimizer and schedule (linear warmup and decay)
no_decay = ['bias', 'LayerNorm.weight']
optimizer_grouped_parameters = [
    {'params': [p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)], 'weight_decay': 0.01},
    {'params': [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)], 'weight_decay': 0.0}
]

optimizer = AdamW(optimizer_grouped_parameters, lr=2e-5, eps=1e-8)
scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps=0, num_training_steps=len(train_loader) * num_epochs)

# Update the training loop to use the scheduler
for epoch in range(num_epochs):
    model.train()
    for batch in train_loader:
        input_ids, attention_mask, labels = [b.to(device) for b in batch]
        
        optimizer.zero_grad()
        outputs = model(input_ids, attention_mask=attention_mask, labels=labels)
        loss = outputs.loss
        loss.backward()
        optimizer.step()
        scheduler.step()
    
    print(f"Epoch {epoch+1}/{num_epochs} completed")

Here's a breakdown of the key components:

  1. Importing the scheduler: The code imports get_linear_schedule_with_warmup from the transformers library.
  2. Optimizer configuration: 
    • - It sets up two groups of parameters: one with weight decay and one without.
    • - This approach helps in applying different learning rates to different parts of the model.
  3. Optimizer and scheduler initialization: 
    • - The AdamW optimizer is initialized with the grouped parameters.
    • - A linear learning rate scheduler with warmup is created, which will adjust the learning rate during training.
  4. Training loop: 
    • - The code updates the training loop to incorporate the scheduler.
    • - After each optimization step, the scheduler is stepped to adjust the learning rate.

This implementation allows for more effective fine-tuning of the BERT model by applying different learning rates to different parts of the model and gradually adjusting the learning rate throughout the training process.

Data Augmentation

To enhance our dataset and potentially improve model performance, we can employ various data augmentation techniques. One particularly effective method is back-translation, which involves translating the original text to another language and then back to the original language. This process introduces subtle variations in the text while preserving its overall meaning and sentiment.

Additionally, we can explore other augmentation strategies such as synonym replacement, random insertion or deletion of words, and text paraphrasing. These techniques collectively help to increase the diversity and size of our training data, potentially leading to a more robust and generalizable sentiment analysis model.

from transformers import MarianMTModel, MarianTokenizer

# Load translation models
en_to_fr = MarianMTModel.from_pretrained('Helsinki-NLP/opus-mt-en-fr')
fr_to_en = MarianMTModel.from_pretrained('Helsinki-NLP/opus-mt-fr-en')
en_tokenizer = MarianTokenizer.from_pretrained('Helsinki-NLP/opus-mt-en-fr')
fr_tokenizer = MarianTokenizer.from_pretrained('Helsinki-NLP/opus-mt-fr-en')

def back_translate(text):
    # Translate to French
    fr_text = en_to_fr.generate(**en_tokenizer(text, return_tensors="pt", padding=True))
    fr_text = [en_tokenizer.decode(t, skip_special_tokens=True) for t in fr_text][0]
    
    # Translate back to English
    en_text = fr_to_en.generate(**fr_tokenizer(fr_text, return_tensors="pt", padding=True))
    en_text = [fr_tokenizer.decode(t, skip_special_tokens=True) for t in en_text][0]
    
    return en_text

# Augment the training data
augmented_texts = [back_translate(text) for text in train_df['text'][:1000]]  # Augment first 1000 samples
augmented_labels = train_df['label'][:1000]

train_df = pd.concat([train_df, pd.DataFrame({'text': augmented_texts, 'label': augmented_labels})])

Here's an explanation of the key components:

  • The code imports necessary models and tokenizers from the Transformers library for translation tasks.
  • It loads pre-trained models for English-to-French and French-to-English translation.
  • The back_translate function is defined to perform the augmentation:
    • It translates the input English text to French
    • Then translates the French text back to English
    • This process introduces subtle variations while preserving the overall meaning
  • The code then augments the training data:
    • It applies back-translation to the first 1000 samples of the training data
    • The augmented texts and their corresponding labels are added to the training dataset

This technique helps increase the diversity of the training data, potentially leading to a more robust and generalizable sentiment analysis model.

Ensemble Methods

To potentially enhance our model's performance and robustness, we can implement an ensemble approach. This technique involves creating multiple models, each with its own strengths and characteristics, and combining their predictions to generate a more accurate and reliable final output.

By leveraging the collective intelligence of various models, we can often achieve better results than relying on a single model alone. This ensemble method can help mitigate individual model weaknesses and capture a broader range of patterns in the data, ultimately leading to improved sentiment analysis accuracy.

# Train multiple models (e.g., BERT, RoBERTa, DistilBERT)
from transformers import RobertaForSequenceClassification, DistilBertForSequenceClassification

models = [
    BertForSequenceClassification.from_pretrained('bert-base-uncased', num_labels=2),
    RobertaForSequenceClassification.from_pretrained('roberta-base', num_labels=2),
    DistilBertForSequenceClassification.from_pretrained('distilbert-base-uncased', num_labels=2)
]

# Train each model (code omitted for brevity)

def ensemble_predict(text):
    encoded = tokenizer.encode_plus(
        text,
        add_special_tokens=True,
        max_length=256,
        padding='max_length',
        truncation=True,
        return_attention_mask=True,
        return_tensors='pt'
    )
    
    input_ids = encoded['input_ids'].to(device)
    attention_mask = encoded['attention_mask'].to(device)
    
    predictions = []
    with torch.no_grad():
        for model in models:
            outputs = model(input_ids, attention_mask=attention_mask)
            pred = torch.softmax(outputs.logits, dim=1)
            predictions.append(pred)
    
    # Average predictions
    avg_pred = torch.mean(torch.stack(predictions), dim=0)
    final_pred = torch.argmax(avg_pred, dim=1)
    
    return "Positive" if final_pred.item() == 1 else "Negative"

This code implements an ensemble method for sentiment analysis using multiple transformer-based models. Here's a breakdown of its key components:

  1. Model Initialization: The code imports and initializes three different pre-trained models: BERT, RoBERTa, and DistilBERT. Each model is set up for binary classification (positive/negative sentiment).
  2. Ensemble Prediction Function: The ensemble_predict function is defined to make predictions using all three models:
    • It tokenizes and encodes the input text using a tokenizer (presumably BERT's tokenizer, though this isn't explicitly shown in the snippet).
    • The encoded input is then passed through each model to get predictions.
    • The raw logits from each model are converted to probabilities using softmax.
    • The predictions from all models are averaged to get a final prediction.
    • The function returns "Positive" or "Negative" based on the averaged prediction.

This ensemble approach aims to improve prediction accuracy by combining the strengths of multiple models, potentially leading to more robust sentiment analysis results.

9.2.7 Conclusion

In this enhanced project, we have successfully implemented a sophisticated sentiment analysis model leveraging the power of BERT architecture. We've delved into advanced techniques to significantly boost its performance and versatility. Our comprehensive approach covered crucial aspects including meticulous data preprocessing, rigorous model training, thorough evaluation, and seamless inference processes.

Furthermore, we've introduced and explored cutting-edge techniques to push the boundaries of model performance. These include the implementation of discriminative learning rates, which allow for fine-tuned optimization across different layers of the model. We've also incorporated data augmentation strategies, particularly back-translation, to enrich our dataset and improve the model's ability to generalize. Additionally, we've ventured into ensemble methods, combining the strengths of multiple models to achieve more robust and accurate predictions.

These advanced techniques serve a dual purpose: they not only potentially enhance the model's accuracy and generalization capabilities but also demonstrate the immense potential of transformer-based models in tackling complex sentiment analysis tasks. By employing these methods, we've showcased how to harness the full power of state-of-the-art NLP architectures, providing a solid and extensible foundation for further exploration and refinement.

The knowledge and experience gained from this project open up numerous avenues for application in real-world NLP scenarios. From analyzing customer feedback and social media sentiment to gauging public opinion on various topics, the techniques explored here have far-reaching implications. This project serves as a springboard for data scientists and NLP practitioners to dive deeper into the fascinating world of sentiment analysis, encouraging further innovation and advancement in this critical area of artificial intelligence and machine learning.

Purchase this book