Chapter 5: Innovations and Challenges in Transformers
5.4 Practical Exercises
This practical exercise section reinforces the concepts discussed in the chapter on Ethical AI by providing hands-on tasks to evaluate, identify, and mitigate biases in language models. Each exercise includes a solution with code to guide implementation.
Exercise 1: Evaluate Gender Bias Using Word Embeddings
Task: Use word embeddings to compare similarities between gendered words and professions to identify potential biases.
Instructions:
- Load a pre-trained word embedding model.
- Compute similarity scores for pairs of gendered words (e.g., "man," "woman") and professions (e.g., "doctor," "nurse").
- Interpret the results to identify potential biases.
Solution:
from whatlies.language import SpacyLanguage
# Load a pretrained language model
language = SpacyLanguage("en_core_web_md")
# Define words
professions = ["doctor", "nurse", "engineer", "teacher"]
genders = ["man", "woman"]
# Calculate and display similarity scores
for profession in professions:
for gender in genders:
similarity = language[profession].similarity(language[gender])
print(f"Similarity between {profession} and {gender}: {similarity:.2f}")Exercise 2: Analyze Model Predictions for Bias
Task: Evaluate a sentiment analysis model's predictions for gender-related sentences.
Instructions:
- Use a Hugging Face sentiment analysis pipeline.
- Provide sentences that vary only by gender (e.g., "He is a doctor." vs. "She is a doctor.").
- Compare the sentiment scores.
Solution:
from transformers import pipeline
# Load sentiment analysis pipeline
sentiment_pipeline = pipeline("sentiment-analysis")
# Test sentences
sentences = [
"He is a doctor.",
"She is a doctor.",
"He is a nurse.",
"She is a nurse."
]
# Perform sentiment analysis
for sentence in sentences:
result = sentiment_pipeline(sentence)
print(f"Sentence: {sentence} => Sentiment: {result[0]['label']}, Score: {result[0]['score']:.2f}")Exercise 3: Mitigate Bias Using Counterfactual Data Augmentation
Task: Create counterfactual examples for a small dataset by flipping gender-related terms.
Instructions:
- Create a dataset with sentences containing gender-specific terms.
- Generate counterfactual examples by replacing "he" with "she" and vice versa.
- Print the augmented dataset.
Solution:
# Original dataset
dataset = ["He is a great leader.", "She is an excellent teacher.", "He solved the problem."]
# Counterfactual augmentation
augmented_dataset = []
for sentence in dataset:
if "He" in sentence:
augmented_dataset.append(sentence.replace("He", "She"))
elif "She" in sentence:
augmented_dataset.append(sentence.replace("She", "He"))
# Combine original and augmented datasets
combined_dataset = dataset + augmented_dataset
print("Augmented Dataset:")
print(combined_dataset)Exercise 4: Evaluate and Visualize Bias Using SHAP
Task: Use SHAP (SHapley Additive exPlanations) to analyze model predictions and visualize potential biases.
Instructions:
- Load a Hugging Face sentiment analysis pipeline.
- Use SHAP to explain predictions for gender-related sentences.
- Visualize the explanations.
Solution:
import shap
from transformers import pipeline
# Load the sentiment analysis pipeline
model_pipeline = pipeline("sentiment-analysis", return_all_scores=True)
# Define sentences for bias analysis
sentences = ["He is a leader.", "She is a leader."]
# Initialize SHAP explainer
explainer = shap.Explainer(model_pipeline)
# Calculate SHAP values
shap_values = explainer(sentences)
# Visualize the SHAP values
shap.plots.text(shap_values)Exercise 5: Filter Dataset for Gender-Neutral Language
Task: Filter a text dataset to remove sentences with gendered pronouns.
Instructions:
- Load a small dataset of sentences.
- Filter out sentences containing "he" or "she."
- Output the filtered dataset.
Solution:
import pandas as pd
# Create a sample dataset
data = pd.DataFrame({
"text": [
"He is a doctor.",
"She is a nurse.",
"Engineers solve problems.",
"Teachers educate students."
]
})
# Filter for gender-neutral sentences
filtered_data = data[~data["text"].str.contains("he|she", case=False)]
print("Filtered Dataset:")
print(filtered_data)These exercises provide practical tools for evaluating and mitigating biases in language models. By completing them, you gain hands-on experience in identifying gender and other biases, understanding their impact, and applying strategies like counterfactual augmentation and dataset filtering to promote fairness. Ethical AI is a continuous effort, and these techniques contribute to building more equitable and inclusive NLP systems.
5.4 Practical Exercises
This practical exercise section reinforces the concepts discussed in the chapter on Ethical AI by providing hands-on tasks to evaluate, identify, and mitigate biases in language models. Each exercise includes a solution with code to guide implementation.
Exercise 1: Evaluate Gender Bias Using Word Embeddings
Task: Use word embeddings to compare similarities between gendered words and professions to identify potential biases.
Instructions:
- Load a pre-trained word embedding model.
- Compute similarity scores for pairs of gendered words (e.g., "man," "woman") and professions (e.g., "doctor," "nurse").
- Interpret the results to identify potential biases.
Solution:
from whatlies.language import SpacyLanguage
# Load a pretrained language model
language = SpacyLanguage("en_core_web_md")
# Define words
professions = ["doctor", "nurse", "engineer", "teacher"]
genders = ["man", "woman"]
# Calculate and display similarity scores
for profession in professions:
for gender in genders:
similarity = language[profession].similarity(language[gender])
print(f"Similarity between {profession} and {gender}: {similarity:.2f}")Exercise 2: Analyze Model Predictions for Bias
Task: Evaluate a sentiment analysis model's predictions for gender-related sentences.
Instructions:
- Use a Hugging Face sentiment analysis pipeline.
- Provide sentences that vary only by gender (e.g., "He is a doctor." vs. "She is a doctor.").
- Compare the sentiment scores.
Solution:
from transformers import pipeline
# Load sentiment analysis pipeline
sentiment_pipeline = pipeline("sentiment-analysis")
# Test sentences
sentences = [
"He is a doctor.",
"She is a doctor.",
"He is a nurse.",
"She is a nurse."
]
# Perform sentiment analysis
for sentence in sentences:
result = sentiment_pipeline(sentence)
print(f"Sentence: {sentence} => Sentiment: {result[0]['label']}, Score: {result[0]['score']:.2f}")Exercise 3: Mitigate Bias Using Counterfactual Data Augmentation
Task: Create counterfactual examples for a small dataset by flipping gender-related terms.
Instructions:
- Create a dataset with sentences containing gender-specific terms.
- Generate counterfactual examples by replacing "he" with "she" and vice versa.
- Print the augmented dataset.
Solution:
# Original dataset
dataset = ["He is a great leader.", "She is an excellent teacher.", "He solved the problem."]
# Counterfactual augmentation
augmented_dataset = []
for sentence in dataset:
if "He" in sentence:
augmented_dataset.append(sentence.replace("He", "She"))
elif "She" in sentence:
augmented_dataset.append(sentence.replace("She", "He"))
# Combine original and augmented datasets
combined_dataset = dataset + augmented_dataset
print("Augmented Dataset:")
print(combined_dataset)Exercise 4: Evaluate and Visualize Bias Using SHAP
Task: Use SHAP (SHapley Additive exPlanations) to analyze model predictions and visualize potential biases.
Instructions:
- Load a Hugging Face sentiment analysis pipeline.
- Use SHAP to explain predictions for gender-related sentences.
- Visualize the explanations.
Solution:
import shap
from transformers import pipeline
# Load the sentiment analysis pipeline
model_pipeline = pipeline("sentiment-analysis", return_all_scores=True)
# Define sentences for bias analysis
sentences = ["He is a leader.", "She is a leader."]
# Initialize SHAP explainer
explainer = shap.Explainer(model_pipeline)
# Calculate SHAP values
shap_values = explainer(sentences)
# Visualize the SHAP values
shap.plots.text(shap_values)Exercise 5: Filter Dataset for Gender-Neutral Language
Task: Filter a text dataset to remove sentences with gendered pronouns.
Instructions:
- Load a small dataset of sentences.
- Filter out sentences containing "he" or "she."
- Output the filtered dataset.
Solution:
import pandas as pd
# Create a sample dataset
data = pd.DataFrame({
"text": [
"He is a doctor.",
"She is a nurse.",
"Engineers solve problems.",
"Teachers educate students."
]
})
# Filter for gender-neutral sentences
filtered_data = data[~data["text"].str.contains("he|she", case=False)]
print("Filtered Dataset:")
print(filtered_data)These exercises provide practical tools for evaluating and mitigating biases in language models. By completing them, you gain hands-on experience in identifying gender and other biases, understanding their impact, and applying strategies like counterfactual augmentation and dataset filtering to promote fairness. Ethical AI is a continuous effort, and these techniques contribute to building more equitable and inclusive NLP systems.
5.4 Practical Exercises
This practical exercise section reinforces the concepts discussed in the chapter on Ethical AI by providing hands-on tasks to evaluate, identify, and mitigate biases in language models. Each exercise includes a solution with code to guide implementation.
Exercise 1: Evaluate Gender Bias Using Word Embeddings
Task: Use word embeddings to compare similarities between gendered words and professions to identify potential biases.
Instructions:
- Load a pre-trained word embedding model.
- Compute similarity scores for pairs of gendered words (e.g., "man," "woman") and professions (e.g., "doctor," "nurse").
- Interpret the results to identify potential biases.
Solution:
from whatlies.language import SpacyLanguage
# Load a pretrained language model
language = SpacyLanguage("en_core_web_md")
# Define words
professions = ["doctor", "nurse", "engineer", "teacher"]
genders = ["man", "woman"]
# Calculate and display similarity scores
for profession in professions:
for gender in genders:
similarity = language[profession].similarity(language[gender])
print(f"Similarity between {profession} and {gender}: {similarity:.2f}")Exercise 2: Analyze Model Predictions for Bias
Task: Evaluate a sentiment analysis model's predictions for gender-related sentences.
Instructions:
- Use a Hugging Face sentiment analysis pipeline.
- Provide sentences that vary only by gender (e.g., "He is a doctor." vs. "She is a doctor.").
- Compare the sentiment scores.
Solution:
from transformers import pipeline
# Load sentiment analysis pipeline
sentiment_pipeline = pipeline("sentiment-analysis")
# Test sentences
sentences = [
"He is a doctor.",
"She is a doctor.",
"He is a nurse.",
"She is a nurse."
]
# Perform sentiment analysis
for sentence in sentences:
result = sentiment_pipeline(sentence)
print(f"Sentence: {sentence} => Sentiment: {result[0]['label']}, Score: {result[0]['score']:.2f}")Exercise 3: Mitigate Bias Using Counterfactual Data Augmentation
Task: Create counterfactual examples for a small dataset by flipping gender-related terms.
Instructions:
- Create a dataset with sentences containing gender-specific terms.
- Generate counterfactual examples by replacing "he" with "she" and vice versa.
- Print the augmented dataset.
Solution:
# Original dataset
dataset = ["He is a great leader.", "She is an excellent teacher.", "He solved the problem."]
# Counterfactual augmentation
augmented_dataset = []
for sentence in dataset:
if "He" in sentence:
augmented_dataset.append(sentence.replace("He", "She"))
elif "She" in sentence:
augmented_dataset.append(sentence.replace("She", "He"))
# Combine original and augmented datasets
combined_dataset = dataset + augmented_dataset
print("Augmented Dataset:")
print(combined_dataset)Exercise 4: Evaluate and Visualize Bias Using SHAP
Task: Use SHAP (SHapley Additive exPlanations) to analyze model predictions and visualize potential biases.
Instructions:
- Load a Hugging Face sentiment analysis pipeline.
- Use SHAP to explain predictions for gender-related sentences.
- Visualize the explanations.
Solution:
import shap
from transformers import pipeline
# Load the sentiment analysis pipeline
model_pipeline = pipeline("sentiment-analysis", return_all_scores=True)
# Define sentences for bias analysis
sentences = ["He is a leader.", "She is a leader."]
# Initialize SHAP explainer
explainer = shap.Explainer(model_pipeline)
# Calculate SHAP values
shap_values = explainer(sentences)
# Visualize the SHAP values
shap.plots.text(shap_values)Exercise 5: Filter Dataset for Gender-Neutral Language
Task: Filter a text dataset to remove sentences with gendered pronouns.
Instructions:
- Load a small dataset of sentences.
- Filter out sentences containing "he" or "she."
- Output the filtered dataset.
Solution:
import pandas as pd
# Create a sample dataset
data = pd.DataFrame({
"text": [
"He is a doctor.",
"She is a nurse.",
"Engineers solve problems.",
"Teachers educate students."
]
})
# Filter for gender-neutral sentences
filtered_data = data[~data["text"].str.contains("he|she", case=False)]
print("Filtered Dataset:")
print(filtered_data)These exercises provide practical tools for evaluating and mitigating biases in language models. By completing them, you gain hands-on experience in identifying gender and other biases, understanding their impact, and applying strategies like counterfactual augmentation and dataset filtering to promote fairness. Ethical AI is a continuous effort, and these techniques contribute to building more equitable and inclusive NLP systems.
5.4 Practical Exercises
This practical exercise section reinforces the concepts discussed in the chapter on Ethical AI by providing hands-on tasks to evaluate, identify, and mitigate biases in language models. Each exercise includes a solution with code to guide implementation.
Exercise 1: Evaluate Gender Bias Using Word Embeddings
Task: Use word embeddings to compare similarities between gendered words and professions to identify potential biases.
Instructions:
- Load a pre-trained word embedding model.
- Compute similarity scores for pairs of gendered words (e.g., "man," "woman") and professions (e.g., "doctor," "nurse").
- Interpret the results to identify potential biases.
Solution:
from whatlies.language import SpacyLanguage
# Load a pretrained language model
language = SpacyLanguage("en_core_web_md")
# Define words
professions = ["doctor", "nurse", "engineer", "teacher"]
genders = ["man", "woman"]
# Calculate and display similarity scores
for profession in professions:
for gender in genders:
similarity = language[profession].similarity(language[gender])
print(f"Similarity between {profession} and {gender}: {similarity:.2f}")Exercise 2: Analyze Model Predictions for Bias
Task: Evaluate a sentiment analysis model's predictions for gender-related sentences.
Instructions:
- Use a Hugging Face sentiment analysis pipeline.
- Provide sentences that vary only by gender (e.g., "He is a doctor." vs. "She is a doctor.").
- Compare the sentiment scores.
Solution:
from transformers import pipeline
# Load sentiment analysis pipeline
sentiment_pipeline = pipeline("sentiment-analysis")
# Test sentences
sentences = [
"He is a doctor.",
"She is a doctor.",
"He is a nurse.",
"She is a nurse."
]
# Perform sentiment analysis
for sentence in sentences:
result = sentiment_pipeline(sentence)
print(f"Sentence: {sentence} => Sentiment: {result[0]['label']}, Score: {result[0]['score']:.2f}")Exercise 3: Mitigate Bias Using Counterfactual Data Augmentation
Task: Create counterfactual examples for a small dataset by flipping gender-related terms.
Instructions:
- Create a dataset with sentences containing gender-specific terms.
- Generate counterfactual examples by replacing "he" with "she" and vice versa.
- Print the augmented dataset.
Solution:
# Original dataset
dataset = ["He is a great leader.", "She is an excellent teacher.", "He solved the problem."]
# Counterfactual augmentation
augmented_dataset = []
for sentence in dataset:
if "He" in sentence:
augmented_dataset.append(sentence.replace("He", "She"))
elif "She" in sentence:
augmented_dataset.append(sentence.replace("She", "He"))
# Combine original and augmented datasets
combined_dataset = dataset + augmented_dataset
print("Augmented Dataset:")
print(combined_dataset)Exercise 4: Evaluate and Visualize Bias Using SHAP
Task: Use SHAP (SHapley Additive exPlanations) to analyze model predictions and visualize potential biases.
Instructions:
- Load a Hugging Face sentiment analysis pipeline.
- Use SHAP to explain predictions for gender-related sentences.
- Visualize the explanations.
Solution:
import shap
from transformers import pipeline
# Load the sentiment analysis pipeline
model_pipeline = pipeline("sentiment-analysis", return_all_scores=True)
# Define sentences for bias analysis
sentences = ["He is a leader.", "She is a leader."]
# Initialize SHAP explainer
explainer = shap.Explainer(model_pipeline)
# Calculate SHAP values
shap_values = explainer(sentences)
# Visualize the SHAP values
shap.plots.text(shap_values)Exercise 5: Filter Dataset for Gender-Neutral Language
Task: Filter a text dataset to remove sentences with gendered pronouns.
Instructions:
- Load a small dataset of sentences.
- Filter out sentences containing "he" or "she."
- Output the filtered dataset.
Solution:
import pandas as pd
# Create a sample dataset
data = pd.DataFrame({
"text": [
"He is a doctor.",
"She is a nurse.",
"Engineers solve problems.",
"Teachers educate students."
]
})
# Filter for gender-neutral sentences
filtered_data = data[~data["text"].str.contains("he|she", case=False)]
print("Filtered Dataset:")
print(filtered_data)These exercises provide practical tools for evaluating and mitigating biases in language models. By completing them, you gain hands-on experience in identifying gender and other biases, understanding their impact, and applying strategies like counterfactual augmentation and dataset filtering to promote fairness. Ethical AI is a continuous effort, and these techniques contribute to building more equitable and inclusive NLP systems.