Chapter 6: Practical Machine Learning Projects
6.1 Project 1: Feature Engineering for Predictive Analytics
This project will focus on applying feature engineering techniques to a dataset to improve the performance of a predictive machine learning model. Feature engineering is crucial for making raw data usable for machine learning algorithms by transforming it into meaningful features that improve model performance.
Project Overview
In this project, we will:
- Explore and preprocess the dataset.
- Apply various feature engineering techniques such as handling missing values, encoding categorical variables, scaling features, and creating new features.
- Build a predictive model using the transformed data to demonstrate the impact of feature engineering on model performance.
- Evaluate the performance of the model before and after feature engineering.
We’ll use the Titanic dataset for this project, as it is well-suited for demonstrating various feature engineering techniques. The task is to predict whether a passenger survived the Titanic disaster based on features like age, gender, ticket class, and fare.
6.1.1 Load and Explore the Dataset
We'll begin by loading the Titanic dataset and conducting a comprehensive initial exploration to gain a thorough understanding of its structure and features. This crucial step involves examining the dataset's dimensions, data types, and basic statistical properties. We'll also investigate the presence of missing values and visualize key relationships between variables to lay a solid foundation for our subsequent feature engineering efforts.
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
# Load the Titanic dataset
url = 'https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv'
titanic_df = pd.read_csv(url)
# Display the first few rows and basic information
print(titanic_df.head())
print(titanic_df.info())
print(titanic_df.describe())
# Visualize missing data
plt.figure(figsize=(10, 6))
sns.heatmap(titanic_df.isnull(), cbar=False, cmap='viridis')
plt.title("Missing Values Heatmap")
plt.show()
# Data Visualization
plt.figure(figsize=(12, 5))
plt.subplot(121)
sns.histplot(titanic_df['Age'].dropna(), kde=True)
plt.title('Age Distribution')
plt.subplot(122)
sns.boxplot(x='Pclass', y='Fare', data=titanic_df)
plt.title('Fare Distribution by Passenger Class')
plt.tight_layout()
plt.show()
# Correlation matrix
# Select only numeric columns for correlation
numeric_cols = titanic_df.select_dtypes(include=['number'])
corr_matrix = numeric_cols.corr()
plt.figure(figsize=(10, 8))
sns.heatmap(corr_matrix, annot=True, cmap='coolwarm')
plt.title('Correlation Matrix')
plt.show()Here's a breakdown of what it does:
- Imports necessary libraries: pandas for data manipulation, matplotlib and seaborn for visualization
- Loads the Titanic dataset from a URL using pandas
- Displays basic information about the dataset:
- First few rows (head())
- Dataset info (info())
- Statistical summary (describe())
- Creates visualizations:
- A heatmap to show missing values in the dataset
- A histogram of the Age distribution
- A boxplot showing Fare distribution by Passenger Class
- A correlation matrix heatmap to show relationships between numerical features
This code is part of the data exploration and preprocessing step, which is crucial for understanding the dataset before applying feature engineering techniques. It helps identify missing data, visualize distributions, and understand relationships between variables, laying the groundwork for subsequent analysis and model building.
6.1.2 Handle Missing Data
The Titanic dataset presents several features with missing values, notably including Age and Cabin. Addressing these missing data points is a crucial step in our feature engineering process.
For the Age feature, we'll employ imputation techniques to fill in the gaps with statistically appropriate values, such as the median age or predictions based on other correlated features. In the case of the Cabin feature, given its high proportion of missing entries, we'll carefully evaluate whether to attempt imputation or to exclude it from our analysis.
This decision will be based on the potential information value of the feature versus the risk of introducing bias through imputation. By systematically handling these missing values, we aim to maximize the usable information in our dataset while maintaining the integrity of our subsequent analyses.
# Fill missing values in the 'Age' column with the median age
titanic_df['Age'].fillna(titanic_df['Age'].median(), inplace=True)
# Fill missing values in the 'Embarked' column with the most frequent value
titanic_df['Embarked'].fillna(titanic_df['Embarked'].mode()[0], inplace=True)
# Drop the 'Cabin' column due to too many missing values
titanic_df.drop(columns=['Cabin'], inplace=True)
print(titanic_df.isnull().sum())Here's a breakdown of what it does:
- It fills missing values in the 'Age' column with the median age of the dataset. This is a common approach for handling missing numerical data.
- For the 'Embarked' column, it fills missing values with the most frequent value (mode) in that column. This is often used for categorical data with missing values.
- The 'Cabin' column is dropped entirely due to having too many missing values. This decision was likely made because the high proportion of missing data in this column could potentially introduce more bias if imputed.
- Finally, it prints the sum of null values in each column after these operations. This helps verify that the missing value handling was successful.
This approach to handling missing data is part of the feature engineering process, aiming to prepare the dataset for machine learning algorithms while preserving as much useful information as possible.
6.1.3 Feature Encoding
The Titanic dataset encompasses several categorical variables, notably Sex and Embarked, which require transformation into a numerical format to be compatible with machine learning algorithms. This conversion process is crucial as most machine learning models are designed to work with numerical inputs. To achieve this transformation, we will employ various encoding techniques, with a particular focus on one-hot encoding.
One-hot encoding is a method that creates binary columns for each category within a categorical variable. For instance, the 'Sex' variable would be split into two columns: 'Sex_male' and 'Sex_female', where each passenger would have a '1' in one column and a '0' in the other. This approach allows us to represent categorical data numerically without implying any ordinal relationship between categories.
Additionally, we may consider other encoding techniques such as label encoding for ordinal variables or target encoding for high-cardinality categorical variables, depending on the specific characteristics of each feature. The choice of encoding method can significantly impact model performance, making it a critical step in our feature engineering process.
# One-hot encode the 'Sex' and 'Embarked' columns
titanic_df = pd.get_dummies(titanic_df, columns=['Sex', 'Embarked'], drop_first=True)
print(titanic_df.head())Here's a breakdown of what it does:
- It uses the
pd.get_dummies()function to one-hot encode the 'Sex' and 'Embarked' columns. - The
columns=['Sex', 'Embarked']parameter specifies which columns to encode. - The
drop_first=Trueargument is used to avoid multicollinearity by dropping one of the created columns for each original categorical variable. - The result is stored back in the
titanic_dfDataFrame, effectively replacing the original 'Sex' and 'Embarked' columns with their one-hot encoded versions. - Finally, it prints the first few rows of the updated DataFrame to show the results of the encoding.
This step is crucial in the feature engineering process as it transforms categorical data into a format that can be easily utilized by machine learning algorithms, which typically require numerical inputs.
6.1.4 Feature Scaling
Feature scaling is a crucial step in our feature engineering process, addressing the significant scale disparities among certain features like Fare and Age. These disparities can have detrimental effects on model performance, particularly for algorithms that are sensitive to feature scales, such as logistic regression or K-nearest neighbors. To mitigate these issues and ensure optimal model performance, we will employ standard scaling as our normalization technique.
Standard scaling, also known as z-score normalization, transforms features to have a mean of 0 and a standard deviation of 1. This transformation preserves the shape of the original distribution while bringing all features to a comparable scale. By applying standard scaling to our dataset, we create a level playing field for all features, allowing algorithms to treat them equally and preventing features with larger magnitudes from dominating the learning process.
The benefits of this scaling approach extend beyond just improving model performance. It also enhances the interpretability of our model coefficients, facilitates faster convergence during the training process, and helps in comparing the relative importance of different features. As we proceed with our analysis, this scaling step will prove instrumental in extracting meaningful insights and building robust predictive models.
from sklearn.preprocessing import StandardScaler
scaling_features = ['Age', 'Fare']
scaler = StandardScaler()
titanic_df[scaling_features] = scaler.fit_transform(titanic_df[scaling_features])
print(titanic_df[scaling_features].head())Here's a breakdown of what it does:
- It imports the StandardScaler class from sklearn.preprocessing.
- It defines a list called 'scaling_features' containing 'Age' and 'Fare', which are the features to be scaled.
- It creates an instance of StandardScaler called 'scaler'.
- It applies the fit_transform method of the scaler to the specified features in the titanic_df DataFrame. This step both fits the scaler to the data and transforms it.
- Finally, it prints the head of the scaled features to show the result.
StandardScaler transforms the features to have a mean of 0 and a standard deviation of 1. This is important for many machine learning algorithms that are sensitive to the scale of input features, as it helps to prevent features with larger magnitudes from dominating the model training process.
6.1.5 Feature Creation
Creating new features is a powerful technique that can significantly enhance a model's ability to capture and leverage complex relationships within the data. This process, known as feature engineering, involves deriving new variables from existing ones to provide additional insights or represent the data in a more meaningful way. In this crucial step of our analysis, we will focus on engineering a new feature called FamilySize.
The FamilySize feature will be created by combining two existing variables: SibSp (number of siblings and spouses aboard) and Parch (number of parents and children aboard). By aggregating these related features, we aim to create a more comprehensive representation of a passenger's family unit size. This new feature has the potential to capture important social dynamics and survival patterns that may not be apparent when considering siblings/spouses and parents/children separately.
The rationale behind this feature engineering decision is rooted in the hypothesis that family size could have played a significant role in survival outcomes during the Titanic disaster. For instance, larger families might have faced different challenges or received different treatment compared to individuals traveling alone or in smaller groups. By creating the FamilySize feature, we provide our model with a more nuanced understanding of each passenger's familial context, potentially improving its predictive capabilities.
# Create a new feature 'FamilySize'
titanic_df['FamilySize'] = titanic_df['SibSp'] + titanic_df['Parch'] + 1
# Create a new feature 'IsAlone'
titanic_df['IsAlone'] = (titanic_df['FamilySize'] == 1).astype(int)
print(titanic_df[['SibSp', 'Parch', 'FamilySize', 'IsAlone']].head())This code snippet demonstrates the creation of two new features in the Titanic dataset through feature engineering:
- FamilySize: This feature is created by summing the values of 'SibSp' (number of siblings and spouses aboard), 'Parch' (number of parents and children aboard), and adding 1 (to include the passenger themselves). This provides a comprehensive measure of the total family size for each passenger.
- IsAlone: This is a binary feature that indicates whether a passenger is traveling alone or not. It's derived from the 'FamilySize' feature, where a value of 1 indicates the passenger is alone, and 0 indicates they are with family.
The code then prints the first few rows of the DataFrame, showing these new features alongside the original 'SibSp' and 'Parch' columns for comparison.
These new features aim to capture more nuanced information about each passenger's family context, which could potentially improve the predictive power of the machine learning model for survival prediction.
6.1.6 Feature Selection
Feature selection is a crucial step in the machine learning pipeline that involves identifying and selecting the most relevant features from the dataset. This process helps to reduce dimensionality, improve model performance, and enhance interpretability. In our Titanic survival prediction project, we'll employ feature selection techniques to identify the most informative features for our predictive model.
There are several methods for feature selection, including filter methods (e.g., correlation-based selection), wrapper methods (e.g., recursive feature elimination), and embedded methods (e.g., L1 regularization). For this project, we'll use a filter method called SelectKBest, which selects features based on their statistical relationship with the target variable.
By applying feature selection, we aim to:
- Reduce overfitting by removing irrelevant or redundant features
- Improve model accuracy by focusing on the most predictive features
- Decrease training time by reducing the dataset's dimensionality
- Enhance model interpretability by identifying the most important features
Let's proceed with implementing the SelectKBest method to choose the top features for our Titanic survival prediction model.
from sklearn.feature_selection import SelectKBest, f_classif
X = titanic_df.drop(columns=['Survived', 'PassengerId', 'Name', 'Ticket'])
y = titanic_df['Survived']
# Select top 10 features
selector = SelectKBest(score_func=f_classif, k=10)
X_selected = selector.fit_transform(X, y)
# Get selected feature names
selected_features = X.columns[selector.get_support()].tolist()
print("Selected features:", selected_features)Here's a breakdown of what the code does:
- It imports the necessary functions from scikit-learn's feature_selection module.
- It prepares the feature matrix X by dropping columns that are not needed for prediction ('Survived', 'PassengerId', 'Name', 'Ticket') from the titanic_df DataFrame.
- It defines the target variable y as the 'Survived' column.
- It creates a SelectKBest object with f_classif as the scoring function and k=10, meaning it will select the top 10 features.
- It applies the feature selection to the data using fit_transform(), which both fits the selector to the data and transforms the data to include only the selected features.
- Finally, it retrieves the names of the selected features and prints them.
This feature selection step is crucial in the machine learning pipeline as it helps to identify the most relevant features for predicting survival on the Titanic. By reducing the number of features to the most informative ones, it can improve model performance, reduce overfitting, and enhance interpretability.
6.1.7 Handle Imbalanced Data
In many real-world datasets, including the Titanic dataset, class imbalance is a common issue. This occurs when one class (in our case, survivors or non-survivors) significantly outnumbers the other. Such imbalance can lead to biased models that perform poorly on the minority class.
To address this issue, we'll use a technique called Synthetic Minority Over-sampling Technique (SMOTE). SMOTE works by creating synthetic examples of the minority class, effectively balancing the dataset. This approach can help improve the model's ability to predict both classes accurately.
from imblearn.over_sampling import SMOTE
# Check class distribution
print("Original class distribution:", y.value_counts())
# Apply SMOTE
smote = SMOTE(random_state=42)
X_resampled, y_resampled = smote.fit_resample(X_selected, y)
print("Resampled class distribution:", pd.Series(y_resampled).value_counts())Here's a breakdown of what the code does:
- It imports the SMOTE class from the imblearn.over_sampling module.
- It prints the original class distribution using y.value_counts() to show the imbalance in the dataset.
- It creates a SMOTE object with a random state of 42 for reproducibility.
- It applies SMOTE to the selected features (X_selected) and the target variable (y) using the fit_resample() method. This creates synthetic examples of the minority class to balance the dataset.
- Finally, it prints the resampled class distribution to show how SMOTE has balanced the classes.
This step is crucial in addressing the class imbalance issue, which can lead to biased models. By creating synthetic examples of the minority class, SMOTE helps improve the model's ability to predict both classes accurately.
6.1.8 Model Building and Evaluation
In this crucial phase of our project, we will construct and assess various machine learning models using the engineered features we've developed. This step is essential for determining the effectiveness of our feature engineering efforts and identifying the most suitable model for predicting Titanic survival.
We'll employ multiple algorithms, including Logistic Regression, Random Forest, and Support Vector Machines (SVM). By comparing their performance, we can gain insights into which model best captures the patterns in our engineered dataset. We'll use cross-validation to ensure robust evaluation and metrics such as accuracy, confusion matrix, and classification report to comprehensively assess each model's performance.
This section will demonstrate how our feature engineering work translates into predictive power, highlighting the importance of the entire process in developing effective machine learning solutions.
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC
from sklearn.metrics import accuracy_score, confusion_matrix, classification_report
# Split the data
X_train, X_test, y_train, y_test = train_test_split(X_resampled, y_resampled, test_size=0.2, random_state=42)
# Initialize models
models = {
'Logistic Regression': LogisticRegression(),
'Random Forest': RandomForestClassifier(),
'SVM': SVC()
}
# Train and evaluate models
for name, model in models.items():
# Cross-validation
cv_scores = cross_val_score(model, X_train, y_train, cv=5)
print(f"{name} CV Score: {cv_scores.mean():.4f} (+/- {cv_scores.std() * 2:.4f})")
# Train the model
model.fit(X_train, y_train)
# Make predictions
y_pred = model.predict(X_test)
# Evaluate the model
print(f"{name} Accuracy: {accuracy_score(y_test, y_pred):.4f}")
print(f"{name} Confusion Matrix:\n", confusion_matrix(y_test, y_pred))
print(f"{name} Classification Report:\n", classification_report(y_test, y_pred))
print("\n")Here's a breakdown of what the code does:
- It imports necessary libraries and functions for model training, evaluation, and cross-validation.
- The data is split into training and testing sets using train_test_split.
- Three different models are initialized: Logistic Regression, Random Forest, and Support Vector Machine (SVM).
- For each model, the code performs the following steps:
- Conducts cross-validation using cross_val_score to assess the model's performance across different subsets of the training data.
- Trains the model on the full training set.
- Makes predictions on the test set.
- Evaluates the model's performance using various metrics:
- Accuracy score
- Confusion matrix
- Classification report (which includes precision, recall, and F1-score)
This comprehensive evaluation allows for a comparison of different models' performance on the engineered features, helping to identify which model best captures the patterns in the dataset. The use of cross-validation ensures a robust evaluation by testing the models on different subsets of the data.
6.1.9 Hyperparameter Tuning
Hyperparameter tuning is a crucial step in optimizing machine learning models. It involves finding the best combination of hyperparameters that yield the highest model performance. In this section, we'll use GridSearchCV to systematically search through a predefined set of hyperparameters for our Random Forest model.
Hyperparameters are parameters that are not learned from the data but are set prior to training. For a Random Forest, these may include the number of trees (n_estimators), the maximum depth of trees (max_depth), and the minimum number of samples required to split an internal node (min_samples_split).
By tuning these hyperparameters, we can potentially improve our model's performance and generalization capabilities. This process helps us find the optimal balance between model complexity and performance, reducing the risk of overfitting or underfitting.
from sklearn.model_selection import GridSearchCV
# Example for Random Forest
param_grid = {
'n_estimators': [100, 200, 300],
'max_depth': [5, 10, None],
'min_samples_split': [2, 5, 10]
}
rf = RandomForestClassifier(random_state=42)
grid_search = GridSearchCV(estimator=rf, param_grid=param_grid, cv=5)
grid_search.fit(X_train, y_train)
print("Best parameters:", grid_search.best_params_)
print("Best cross-validation score:", grid_search.best_score_)
# Evaluate the best model
best_model = grid_search.best_estimator_
y_pred = best_model.predict(X_test)
print("Best Model Accuracy:", accuracy_score(y_test, y_pred))Here's a breakdown of what the code does:
- It imports GridSearchCV from scikit-learn, which is used for searching the best parameters for a model.
- A parameter grid (param_grid) is defined with different values for 'n_estimators', 'max_depth', and 'min_samples_split'. These are the hyperparameters we want to optimize.
- A RandomForestClassifier is initialized with a fixed random state for reproducibility.
- GridSearchCV is set up with the Random Forest model, the parameter grid, and 5-fold cross-validation.
- The grid search is performed using fit() on the training data.
- The best parameters and the best cross-validation score are printed.
- Finally, the best model (with optimized parameters) is used to make predictions on the test set, and its accuracy is printed.
This process helps find the optimal combination of hyperparameters that yields the best model performance, potentially improving the model's accuracy and generalization capabilities.
6.1.10 Feature Importance Analysis
Feature importance analysis is a crucial step in understanding which features contribute most significantly to our model's predictions. This analysis helps us identify the most influential factors in determining Titanic passenger survival, providing valuable insights into the dataset and our model's decision-making process.
By examining feature importance, we can:
- Gain a deeper understanding of the factors that most affected survival rates
- Validate our feature engineering efforts by seeing which engineered features are most impactful
- Potentially simplify our model by focusing on the most important features
- Inform future data collection efforts by highlighting the most critical information
In the following code, we'll use our best Random Forest model to calculate and visualize feature importance, providing a clear picture of which features are driving our predictions.
# Using the best Random Forest model
feature_importance = best_model.feature_importances_
feature_names = X.columns[selector.get_support()].tolist()
# Sort features by importance
feature_importance_sorted = sorted(zip(feature_importance, feature_names), reverse=True)
# Plot feature importance
plt.figure(figsize=(10, 6))
plt.bar([x[1] for x in feature_importance_sorted], [x[0] for x in feature_importance_sorted])
plt.title('Feature Importance')
plt.xlabel('Features')
plt.ylabel('Importance')
plt.xticks(rotation=45)
plt.tight_layout()
plt.show()Here's a breakdown of what the code does:
- It extracts feature importance scores from the best Random Forest model using
best_model.feature_importances_. - It retrieves the names of the selected features using
X.columns[selector.get_support()].tolist(). - The feature importances and names are combined and sorted in descending order of importance.
- A bar plot is created to visualize the feature importances:
- The plot is set to a size of 10x6 inches.
- Feature names are placed on the x-axis and their importance scores on the y-axis.
- The plot is given a title, x-label, and y-label.
- X-axis labels are rotated 45 degrees for better readability.
This visualization helps identify which features have the most significant impact on the model's predictions, providing insights into the factors that most influence survival predictions in the Titanic dataset.
6.1.11 Error Analysis
Error analysis is a crucial step in understanding where our model is making mistakes and why. This process involves examining the instances where the model's predictions differ from the actual outcomes. By analyzing these misclassifications, we can gain valuable insights into our model's weaknesses and identify potential areas for improvement.
In this section, we'll look at the characteristics of misclassified samples, comparing their features to those of correctly classified instances. This analysis can reveal patterns or specific subgroups where the model struggles, potentially highlighting the need for additional feature engineering, data collection, or model adjustments.
import pandas as pd
# Convert X_test to DataFrame with column names
X_test_df = pd.DataFrame(X_test, columns=selected_features)
# Identify misclassified samples
misclassified = X_test_df[y_test != y_pred].copy()
misclassified['true_label'] = y_test[y_test != y_pred]
misclassified['predicted_label'] = y_pred[y_test != y_pred]
# Display sample misclassified instances
print("Sample of misclassified instances:")
print(misclassified.head())
# Analyze misclassifications
print("\nMisclassification analysis:")
for feature in selected_features:
print(f"\nFeature: {feature}")
print(misclassified.groupby(['true_label', 'predicted_label'])[feature].mean())Here's an explanation of what the code does:
- It identifies misclassified samples by comparing the actual labels (
y_test) with the predicted labels (y_pred). - It creates a new DataFrame called
misclassified, containing only the incorrectly classified instances from the test set. - It adds two new columns to this DataFrame:
'true_label': the actual label fromy_test'predicted_label': the label predicted by the model (y_pred)
- It prints a sample of these misclassified instances using the
head()function. - Then, it performs a detailed analysis of the misclassifications:
- It iterates through each feature in the
misclassifiedDataFrame. - For each feature, it calculates and prints the mean value grouped by
true_labelandpredicted_label. - This helps in understanding patterns in the model’s mistakes.
- It iterates through each feature in the
Why is this useful?
- It allows us to pinpoint specific features where misclassification occurs.
- It helps identify potential biases in the model.
- It can guide feature engineering improvements or hyperparameter tuning to enhance the model’s performance.
6.1.12 Conclusion
In this project, we applied various feature engineering techniques to the Titanic dataset and built multiple predictive models. We expanded on the original project by including data visualization, feature selection, handling imbalanced data, trying multiple models, implementing cross-validation, performing hyperparameter tuning, analyzing feature importance, and conducting error analysis. These additional steps provide a more comprehensive understanding of the dataset and the impact of feature engineering on model performance.
The results demonstrate the importance of feature engineering in improving model accuracy and interpretability. By carefully selecting, transforming, and creating features, we were able to build more robust predictive models. The feature importance analysis and error analysis provide insights into which factors are most crucial for predicting survival and where the model might be falling short.
This project serves as a thorough example of the feature engineering process and its significance in the machine learning pipeline. It showcases how various techniques can be combined to extract meaningful insights from raw data and improve model performance.
6.1 Project 1: Feature Engineering for Predictive Analytics
This project will focus on applying feature engineering techniques to a dataset to improve the performance of a predictive machine learning model. Feature engineering is crucial for making raw data usable for machine learning algorithms by transforming it into meaningful features that improve model performance.
Project Overview
In this project, we will:
- Explore and preprocess the dataset.
- Apply various feature engineering techniques such as handling missing values, encoding categorical variables, scaling features, and creating new features.
- Build a predictive model using the transformed data to demonstrate the impact of feature engineering on model performance.
- Evaluate the performance of the model before and after feature engineering.
We’ll use the Titanic dataset for this project, as it is well-suited for demonstrating various feature engineering techniques. The task is to predict whether a passenger survived the Titanic disaster based on features like age, gender, ticket class, and fare.
6.1.1 Load and Explore the Dataset
We'll begin by loading the Titanic dataset and conducting a comprehensive initial exploration to gain a thorough understanding of its structure and features. This crucial step involves examining the dataset's dimensions, data types, and basic statistical properties. We'll also investigate the presence of missing values and visualize key relationships between variables to lay a solid foundation for our subsequent feature engineering efforts.
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
# Load the Titanic dataset
url = 'https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv'
titanic_df = pd.read_csv(url)
# Display the first few rows and basic information
print(titanic_df.head())
print(titanic_df.info())
print(titanic_df.describe())
# Visualize missing data
plt.figure(figsize=(10, 6))
sns.heatmap(titanic_df.isnull(), cbar=False, cmap='viridis')
plt.title("Missing Values Heatmap")
plt.show()
# Data Visualization
plt.figure(figsize=(12, 5))
plt.subplot(121)
sns.histplot(titanic_df['Age'].dropna(), kde=True)
plt.title('Age Distribution')
plt.subplot(122)
sns.boxplot(x='Pclass', y='Fare', data=titanic_df)
plt.title('Fare Distribution by Passenger Class')
plt.tight_layout()
plt.show()
# Correlation matrix
# Select only numeric columns for correlation
numeric_cols = titanic_df.select_dtypes(include=['number'])
corr_matrix = numeric_cols.corr()
plt.figure(figsize=(10, 8))
sns.heatmap(corr_matrix, annot=True, cmap='coolwarm')
plt.title('Correlation Matrix')
plt.show()Here's a breakdown of what it does:
- Imports necessary libraries: pandas for data manipulation, matplotlib and seaborn for visualization
- Loads the Titanic dataset from a URL using pandas
- Displays basic information about the dataset:
- First few rows (head())
- Dataset info (info())
- Statistical summary (describe())
- Creates visualizations:
- A heatmap to show missing values in the dataset
- A histogram of the Age distribution
- A boxplot showing Fare distribution by Passenger Class
- A correlation matrix heatmap to show relationships between numerical features
This code is part of the data exploration and preprocessing step, which is crucial for understanding the dataset before applying feature engineering techniques. It helps identify missing data, visualize distributions, and understand relationships between variables, laying the groundwork for subsequent analysis and model building.
6.1.2 Handle Missing Data
The Titanic dataset presents several features with missing values, notably including Age and Cabin. Addressing these missing data points is a crucial step in our feature engineering process.
For the Age feature, we'll employ imputation techniques to fill in the gaps with statistically appropriate values, such as the median age or predictions based on other correlated features. In the case of the Cabin feature, given its high proportion of missing entries, we'll carefully evaluate whether to attempt imputation or to exclude it from our analysis.
This decision will be based on the potential information value of the feature versus the risk of introducing bias through imputation. By systematically handling these missing values, we aim to maximize the usable information in our dataset while maintaining the integrity of our subsequent analyses.
# Fill missing values in the 'Age' column with the median age
titanic_df['Age'].fillna(titanic_df['Age'].median(), inplace=True)
# Fill missing values in the 'Embarked' column with the most frequent value
titanic_df['Embarked'].fillna(titanic_df['Embarked'].mode()[0], inplace=True)
# Drop the 'Cabin' column due to too many missing values
titanic_df.drop(columns=['Cabin'], inplace=True)
print(titanic_df.isnull().sum())Here's a breakdown of what it does:
- It fills missing values in the 'Age' column with the median age of the dataset. This is a common approach for handling missing numerical data.
- For the 'Embarked' column, it fills missing values with the most frequent value (mode) in that column. This is often used for categorical data with missing values.
- The 'Cabin' column is dropped entirely due to having too many missing values. This decision was likely made because the high proportion of missing data in this column could potentially introduce more bias if imputed.
- Finally, it prints the sum of null values in each column after these operations. This helps verify that the missing value handling was successful.
This approach to handling missing data is part of the feature engineering process, aiming to prepare the dataset for machine learning algorithms while preserving as much useful information as possible.
6.1.3 Feature Encoding
The Titanic dataset encompasses several categorical variables, notably Sex and Embarked, which require transformation into a numerical format to be compatible with machine learning algorithms. This conversion process is crucial as most machine learning models are designed to work with numerical inputs. To achieve this transformation, we will employ various encoding techniques, with a particular focus on one-hot encoding.
One-hot encoding is a method that creates binary columns for each category within a categorical variable. For instance, the 'Sex' variable would be split into two columns: 'Sex_male' and 'Sex_female', where each passenger would have a '1' in one column and a '0' in the other. This approach allows us to represent categorical data numerically without implying any ordinal relationship between categories.
Additionally, we may consider other encoding techniques such as label encoding for ordinal variables or target encoding for high-cardinality categorical variables, depending on the specific characteristics of each feature. The choice of encoding method can significantly impact model performance, making it a critical step in our feature engineering process.
# One-hot encode the 'Sex' and 'Embarked' columns
titanic_df = pd.get_dummies(titanic_df, columns=['Sex', 'Embarked'], drop_first=True)
print(titanic_df.head())Here's a breakdown of what it does:
- It uses the
pd.get_dummies()function to one-hot encode the 'Sex' and 'Embarked' columns. - The
columns=['Sex', 'Embarked']parameter specifies which columns to encode. - The
drop_first=Trueargument is used to avoid multicollinearity by dropping one of the created columns for each original categorical variable. - The result is stored back in the
titanic_dfDataFrame, effectively replacing the original 'Sex' and 'Embarked' columns with their one-hot encoded versions. - Finally, it prints the first few rows of the updated DataFrame to show the results of the encoding.
This step is crucial in the feature engineering process as it transforms categorical data into a format that can be easily utilized by machine learning algorithms, which typically require numerical inputs.
6.1.4 Feature Scaling
Feature scaling is a crucial step in our feature engineering process, addressing the significant scale disparities among certain features like Fare and Age. These disparities can have detrimental effects on model performance, particularly for algorithms that are sensitive to feature scales, such as logistic regression or K-nearest neighbors. To mitigate these issues and ensure optimal model performance, we will employ standard scaling as our normalization technique.
Standard scaling, also known as z-score normalization, transforms features to have a mean of 0 and a standard deviation of 1. This transformation preserves the shape of the original distribution while bringing all features to a comparable scale. By applying standard scaling to our dataset, we create a level playing field for all features, allowing algorithms to treat them equally and preventing features with larger magnitudes from dominating the learning process.
The benefits of this scaling approach extend beyond just improving model performance. It also enhances the interpretability of our model coefficients, facilitates faster convergence during the training process, and helps in comparing the relative importance of different features. As we proceed with our analysis, this scaling step will prove instrumental in extracting meaningful insights and building robust predictive models.
from sklearn.preprocessing import StandardScaler
scaling_features = ['Age', 'Fare']
scaler = StandardScaler()
titanic_df[scaling_features] = scaler.fit_transform(titanic_df[scaling_features])
print(titanic_df[scaling_features].head())Here's a breakdown of what it does:
- It imports the StandardScaler class from sklearn.preprocessing.
- It defines a list called 'scaling_features' containing 'Age' and 'Fare', which are the features to be scaled.
- It creates an instance of StandardScaler called 'scaler'.
- It applies the fit_transform method of the scaler to the specified features in the titanic_df DataFrame. This step both fits the scaler to the data and transforms it.
- Finally, it prints the head of the scaled features to show the result.
StandardScaler transforms the features to have a mean of 0 and a standard deviation of 1. This is important for many machine learning algorithms that are sensitive to the scale of input features, as it helps to prevent features with larger magnitudes from dominating the model training process.
6.1.5 Feature Creation
Creating new features is a powerful technique that can significantly enhance a model's ability to capture and leverage complex relationships within the data. This process, known as feature engineering, involves deriving new variables from existing ones to provide additional insights or represent the data in a more meaningful way. In this crucial step of our analysis, we will focus on engineering a new feature called FamilySize.
The FamilySize feature will be created by combining two existing variables: SibSp (number of siblings and spouses aboard) and Parch (number of parents and children aboard). By aggregating these related features, we aim to create a more comprehensive representation of a passenger's family unit size. This new feature has the potential to capture important social dynamics and survival patterns that may not be apparent when considering siblings/spouses and parents/children separately.
The rationale behind this feature engineering decision is rooted in the hypothesis that family size could have played a significant role in survival outcomes during the Titanic disaster. For instance, larger families might have faced different challenges or received different treatment compared to individuals traveling alone or in smaller groups. By creating the FamilySize feature, we provide our model with a more nuanced understanding of each passenger's familial context, potentially improving its predictive capabilities.
# Create a new feature 'FamilySize'
titanic_df['FamilySize'] = titanic_df['SibSp'] + titanic_df['Parch'] + 1
# Create a new feature 'IsAlone'
titanic_df['IsAlone'] = (titanic_df['FamilySize'] == 1).astype(int)
print(titanic_df[['SibSp', 'Parch', 'FamilySize', 'IsAlone']].head())This code snippet demonstrates the creation of two new features in the Titanic dataset through feature engineering:
- FamilySize: This feature is created by summing the values of 'SibSp' (number of siblings and spouses aboard), 'Parch' (number of parents and children aboard), and adding 1 (to include the passenger themselves). This provides a comprehensive measure of the total family size for each passenger.
- IsAlone: This is a binary feature that indicates whether a passenger is traveling alone or not. It's derived from the 'FamilySize' feature, where a value of 1 indicates the passenger is alone, and 0 indicates they are with family.
The code then prints the first few rows of the DataFrame, showing these new features alongside the original 'SibSp' and 'Parch' columns for comparison.
These new features aim to capture more nuanced information about each passenger's family context, which could potentially improve the predictive power of the machine learning model for survival prediction.
6.1.6 Feature Selection
Feature selection is a crucial step in the machine learning pipeline that involves identifying and selecting the most relevant features from the dataset. This process helps to reduce dimensionality, improve model performance, and enhance interpretability. In our Titanic survival prediction project, we'll employ feature selection techniques to identify the most informative features for our predictive model.
There are several methods for feature selection, including filter methods (e.g., correlation-based selection), wrapper methods (e.g., recursive feature elimination), and embedded methods (e.g., L1 regularization). For this project, we'll use a filter method called SelectKBest, which selects features based on their statistical relationship with the target variable.
By applying feature selection, we aim to:
- Reduce overfitting by removing irrelevant or redundant features
- Improve model accuracy by focusing on the most predictive features
- Decrease training time by reducing the dataset's dimensionality
- Enhance model interpretability by identifying the most important features
Let's proceed with implementing the SelectKBest method to choose the top features for our Titanic survival prediction model.
from sklearn.feature_selection import SelectKBest, f_classif
X = titanic_df.drop(columns=['Survived', 'PassengerId', 'Name', 'Ticket'])
y = titanic_df['Survived']
# Select top 10 features
selector = SelectKBest(score_func=f_classif, k=10)
X_selected = selector.fit_transform(X, y)
# Get selected feature names
selected_features = X.columns[selector.get_support()].tolist()
print("Selected features:", selected_features)Here's a breakdown of what the code does:
- It imports the necessary functions from scikit-learn's feature_selection module.
- It prepares the feature matrix X by dropping columns that are not needed for prediction ('Survived', 'PassengerId', 'Name', 'Ticket') from the titanic_df DataFrame.
- It defines the target variable y as the 'Survived' column.
- It creates a SelectKBest object with f_classif as the scoring function and k=10, meaning it will select the top 10 features.
- It applies the feature selection to the data using fit_transform(), which both fits the selector to the data and transforms the data to include only the selected features.
- Finally, it retrieves the names of the selected features and prints them.
This feature selection step is crucial in the machine learning pipeline as it helps to identify the most relevant features for predicting survival on the Titanic. By reducing the number of features to the most informative ones, it can improve model performance, reduce overfitting, and enhance interpretability.
6.1.7 Handle Imbalanced Data
In many real-world datasets, including the Titanic dataset, class imbalance is a common issue. This occurs when one class (in our case, survivors or non-survivors) significantly outnumbers the other. Such imbalance can lead to biased models that perform poorly on the minority class.
To address this issue, we'll use a technique called Synthetic Minority Over-sampling Technique (SMOTE). SMOTE works by creating synthetic examples of the minority class, effectively balancing the dataset. This approach can help improve the model's ability to predict both classes accurately.
from imblearn.over_sampling import SMOTE
# Check class distribution
print("Original class distribution:", y.value_counts())
# Apply SMOTE
smote = SMOTE(random_state=42)
X_resampled, y_resampled = smote.fit_resample(X_selected, y)
print("Resampled class distribution:", pd.Series(y_resampled).value_counts())Here's a breakdown of what the code does:
- It imports the SMOTE class from the imblearn.over_sampling module.
- It prints the original class distribution using y.value_counts() to show the imbalance in the dataset.
- It creates a SMOTE object with a random state of 42 for reproducibility.
- It applies SMOTE to the selected features (X_selected) and the target variable (y) using the fit_resample() method. This creates synthetic examples of the minority class to balance the dataset.
- Finally, it prints the resampled class distribution to show how SMOTE has balanced the classes.
This step is crucial in addressing the class imbalance issue, which can lead to biased models. By creating synthetic examples of the minority class, SMOTE helps improve the model's ability to predict both classes accurately.
6.1.8 Model Building and Evaluation
In this crucial phase of our project, we will construct and assess various machine learning models using the engineered features we've developed. This step is essential for determining the effectiveness of our feature engineering efforts and identifying the most suitable model for predicting Titanic survival.
We'll employ multiple algorithms, including Logistic Regression, Random Forest, and Support Vector Machines (SVM). By comparing their performance, we can gain insights into which model best captures the patterns in our engineered dataset. We'll use cross-validation to ensure robust evaluation and metrics such as accuracy, confusion matrix, and classification report to comprehensively assess each model's performance.
This section will demonstrate how our feature engineering work translates into predictive power, highlighting the importance of the entire process in developing effective machine learning solutions.
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC
from sklearn.metrics import accuracy_score, confusion_matrix, classification_report
# Split the data
X_train, X_test, y_train, y_test = train_test_split(X_resampled, y_resampled, test_size=0.2, random_state=42)
# Initialize models
models = {
'Logistic Regression': LogisticRegression(),
'Random Forest': RandomForestClassifier(),
'SVM': SVC()
}
# Train and evaluate models
for name, model in models.items():
# Cross-validation
cv_scores = cross_val_score(model, X_train, y_train, cv=5)
print(f"{name} CV Score: {cv_scores.mean():.4f} (+/- {cv_scores.std() * 2:.4f})")
# Train the model
model.fit(X_train, y_train)
# Make predictions
y_pred = model.predict(X_test)
# Evaluate the model
print(f"{name} Accuracy: {accuracy_score(y_test, y_pred):.4f}")
print(f"{name} Confusion Matrix:\n", confusion_matrix(y_test, y_pred))
print(f"{name} Classification Report:\n", classification_report(y_test, y_pred))
print("\n")Here's a breakdown of what the code does:
- It imports necessary libraries and functions for model training, evaluation, and cross-validation.
- The data is split into training and testing sets using train_test_split.
- Three different models are initialized: Logistic Regression, Random Forest, and Support Vector Machine (SVM).
- For each model, the code performs the following steps:
- Conducts cross-validation using cross_val_score to assess the model's performance across different subsets of the training data.
- Trains the model on the full training set.
- Makes predictions on the test set.
- Evaluates the model's performance using various metrics:
- Accuracy score
- Confusion matrix
- Classification report (which includes precision, recall, and F1-score)
This comprehensive evaluation allows for a comparison of different models' performance on the engineered features, helping to identify which model best captures the patterns in the dataset. The use of cross-validation ensures a robust evaluation by testing the models on different subsets of the data.
6.1.9 Hyperparameter Tuning
Hyperparameter tuning is a crucial step in optimizing machine learning models. It involves finding the best combination of hyperparameters that yield the highest model performance. In this section, we'll use GridSearchCV to systematically search through a predefined set of hyperparameters for our Random Forest model.
Hyperparameters are parameters that are not learned from the data but are set prior to training. For a Random Forest, these may include the number of trees (n_estimators), the maximum depth of trees (max_depth), and the minimum number of samples required to split an internal node (min_samples_split).
By tuning these hyperparameters, we can potentially improve our model's performance and generalization capabilities. This process helps us find the optimal balance between model complexity and performance, reducing the risk of overfitting or underfitting.
from sklearn.model_selection import GridSearchCV
# Example for Random Forest
param_grid = {
'n_estimators': [100, 200, 300],
'max_depth': [5, 10, None],
'min_samples_split': [2, 5, 10]
}
rf = RandomForestClassifier(random_state=42)
grid_search = GridSearchCV(estimator=rf, param_grid=param_grid, cv=5)
grid_search.fit(X_train, y_train)
print("Best parameters:", grid_search.best_params_)
print("Best cross-validation score:", grid_search.best_score_)
# Evaluate the best model
best_model = grid_search.best_estimator_
y_pred = best_model.predict(X_test)
print("Best Model Accuracy:", accuracy_score(y_test, y_pred))Here's a breakdown of what the code does:
- It imports GridSearchCV from scikit-learn, which is used for searching the best parameters for a model.
- A parameter grid (param_grid) is defined with different values for 'n_estimators', 'max_depth', and 'min_samples_split'. These are the hyperparameters we want to optimize.
- A RandomForestClassifier is initialized with a fixed random state for reproducibility.
- GridSearchCV is set up with the Random Forest model, the parameter grid, and 5-fold cross-validation.
- The grid search is performed using fit() on the training data.
- The best parameters and the best cross-validation score are printed.
- Finally, the best model (with optimized parameters) is used to make predictions on the test set, and its accuracy is printed.
This process helps find the optimal combination of hyperparameters that yields the best model performance, potentially improving the model's accuracy and generalization capabilities.
6.1.10 Feature Importance Analysis
Feature importance analysis is a crucial step in understanding which features contribute most significantly to our model's predictions. This analysis helps us identify the most influential factors in determining Titanic passenger survival, providing valuable insights into the dataset and our model's decision-making process.
By examining feature importance, we can:
- Gain a deeper understanding of the factors that most affected survival rates
- Validate our feature engineering efforts by seeing which engineered features are most impactful
- Potentially simplify our model by focusing on the most important features
- Inform future data collection efforts by highlighting the most critical information
In the following code, we'll use our best Random Forest model to calculate and visualize feature importance, providing a clear picture of which features are driving our predictions.
# Using the best Random Forest model
feature_importance = best_model.feature_importances_
feature_names = X.columns[selector.get_support()].tolist()
# Sort features by importance
feature_importance_sorted = sorted(zip(feature_importance, feature_names), reverse=True)
# Plot feature importance
plt.figure(figsize=(10, 6))
plt.bar([x[1] for x in feature_importance_sorted], [x[0] for x in feature_importance_sorted])
plt.title('Feature Importance')
plt.xlabel('Features')
plt.ylabel('Importance')
plt.xticks(rotation=45)
plt.tight_layout()
plt.show()Here's a breakdown of what the code does:
- It extracts feature importance scores from the best Random Forest model using
best_model.feature_importances_. - It retrieves the names of the selected features using
X.columns[selector.get_support()].tolist(). - The feature importances and names are combined and sorted in descending order of importance.
- A bar plot is created to visualize the feature importances:
- The plot is set to a size of 10x6 inches.
- Feature names are placed on the x-axis and their importance scores on the y-axis.
- The plot is given a title, x-label, and y-label.
- X-axis labels are rotated 45 degrees for better readability.
This visualization helps identify which features have the most significant impact on the model's predictions, providing insights into the factors that most influence survival predictions in the Titanic dataset.
6.1.11 Error Analysis
Error analysis is a crucial step in understanding where our model is making mistakes and why. This process involves examining the instances where the model's predictions differ from the actual outcomes. By analyzing these misclassifications, we can gain valuable insights into our model's weaknesses and identify potential areas for improvement.
In this section, we'll look at the characteristics of misclassified samples, comparing their features to those of correctly classified instances. This analysis can reveal patterns or specific subgroups where the model struggles, potentially highlighting the need for additional feature engineering, data collection, or model adjustments.
import pandas as pd
# Convert X_test to DataFrame with column names
X_test_df = pd.DataFrame(X_test, columns=selected_features)
# Identify misclassified samples
misclassified = X_test_df[y_test != y_pred].copy()
misclassified['true_label'] = y_test[y_test != y_pred]
misclassified['predicted_label'] = y_pred[y_test != y_pred]
# Display sample misclassified instances
print("Sample of misclassified instances:")
print(misclassified.head())
# Analyze misclassifications
print("\nMisclassification analysis:")
for feature in selected_features:
print(f"\nFeature: {feature}")
print(misclassified.groupby(['true_label', 'predicted_label'])[feature].mean())Here's an explanation of what the code does:
- It identifies misclassified samples by comparing the actual labels (
y_test) with the predicted labels (y_pred). - It creates a new DataFrame called
misclassified, containing only the incorrectly classified instances from the test set. - It adds two new columns to this DataFrame:
'true_label': the actual label fromy_test'predicted_label': the label predicted by the model (y_pred)
- It prints a sample of these misclassified instances using the
head()function. - Then, it performs a detailed analysis of the misclassifications:
- It iterates through each feature in the
misclassifiedDataFrame. - For each feature, it calculates and prints the mean value grouped by
true_labelandpredicted_label. - This helps in understanding patterns in the model’s mistakes.
- It iterates through each feature in the
Why is this useful?
- It allows us to pinpoint specific features where misclassification occurs.
- It helps identify potential biases in the model.
- It can guide feature engineering improvements or hyperparameter tuning to enhance the model’s performance.
6.1.12 Conclusion
In this project, we applied various feature engineering techniques to the Titanic dataset and built multiple predictive models. We expanded on the original project by including data visualization, feature selection, handling imbalanced data, trying multiple models, implementing cross-validation, performing hyperparameter tuning, analyzing feature importance, and conducting error analysis. These additional steps provide a more comprehensive understanding of the dataset and the impact of feature engineering on model performance.
The results demonstrate the importance of feature engineering in improving model accuracy and interpretability. By carefully selecting, transforming, and creating features, we were able to build more robust predictive models. The feature importance analysis and error analysis provide insights into which factors are most crucial for predicting survival and where the model might be falling short.
This project serves as a thorough example of the feature engineering process and its significance in the machine learning pipeline. It showcases how various techniques can be combined to extract meaningful insights from raw data and improve model performance.
6.1 Project 1: Feature Engineering for Predictive Analytics
This project will focus on applying feature engineering techniques to a dataset to improve the performance of a predictive machine learning model. Feature engineering is crucial for making raw data usable for machine learning algorithms by transforming it into meaningful features that improve model performance.
Project Overview
In this project, we will:
- Explore and preprocess the dataset.
- Apply various feature engineering techniques such as handling missing values, encoding categorical variables, scaling features, and creating new features.
- Build a predictive model using the transformed data to demonstrate the impact of feature engineering on model performance.
- Evaluate the performance of the model before and after feature engineering.
We’ll use the Titanic dataset for this project, as it is well-suited for demonstrating various feature engineering techniques. The task is to predict whether a passenger survived the Titanic disaster based on features like age, gender, ticket class, and fare.
6.1.1 Load and Explore the Dataset
We'll begin by loading the Titanic dataset and conducting a comprehensive initial exploration to gain a thorough understanding of its structure and features. This crucial step involves examining the dataset's dimensions, data types, and basic statistical properties. We'll also investigate the presence of missing values and visualize key relationships between variables to lay a solid foundation for our subsequent feature engineering efforts.
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
# Load the Titanic dataset
url = 'https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv'
titanic_df = pd.read_csv(url)
# Display the first few rows and basic information
print(titanic_df.head())
print(titanic_df.info())
print(titanic_df.describe())
# Visualize missing data
plt.figure(figsize=(10, 6))
sns.heatmap(titanic_df.isnull(), cbar=False, cmap='viridis')
plt.title("Missing Values Heatmap")
plt.show()
# Data Visualization
plt.figure(figsize=(12, 5))
plt.subplot(121)
sns.histplot(titanic_df['Age'].dropna(), kde=True)
plt.title('Age Distribution')
plt.subplot(122)
sns.boxplot(x='Pclass', y='Fare', data=titanic_df)
plt.title('Fare Distribution by Passenger Class')
plt.tight_layout()
plt.show()
# Correlation matrix
# Select only numeric columns for correlation
numeric_cols = titanic_df.select_dtypes(include=['number'])
corr_matrix = numeric_cols.corr()
plt.figure(figsize=(10, 8))
sns.heatmap(corr_matrix, annot=True, cmap='coolwarm')
plt.title('Correlation Matrix')
plt.show()Here's a breakdown of what it does:
- Imports necessary libraries: pandas for data manipulation, matplotlib and seaborn for visualization
- Loads the Titanic dataset from a URL using pandas
- Displays basic information about the dataset:
- First few rows (head())
- Dataset info (info())
- Statistical summary (describe())
- Creates visualizations:
- A heatmap to show missing values in the dataset
- A histogram of the Age distribution
- A boxplot showing Fare distribution by Passenger Class
- A correlation matrix heatmap to show relationships between numerical features
This code is part of the data exploration and preprocessing step, which is crucial for understanding the dataset before applying feature engineering techniques. It helps identify missing data, visualize distributions, and understand relationships between variables, laying the groundwork for subsequent analysis and model building.
6.1.2 Handle Missing Data
The Titanic dataset presents several features with missing values, notably including Age and Cabin. Addressing these missing data points is a crucial step in our feature engineering process.
For the Age feature, we'll employ imputation techniques to fill in the gaps with statistically appropriate values, such as the median age or predictions based on other correlated features. In the case of the Cabin feature, given its high proportion of missing entries, we'll carefully evaluate whether to attempt imputation or to exclude it from our analysis.
This decision will be based on the potential information value of the feature versus the risk of introducing bias through imputation. By systematically handling these missing values, we aim to maximize the usable information in our dataset while maintaining the integrity of our subsequent analyses.
# Fill missing values in the 'Age' column with the median age
titanic_df['Age'].fillna(titanic_df['Age'].median(), inplace=True)
# Fill missing values in the 'Embarked' column with the most frequent value
titanic_df['Embarked'].fillna(titanic_df['Embarked'].mode()[0], inplace=True)
# Drop the 'Cabin' column due to too many missing values
titanic_df.drop(columns=['Cabin'], inplace=True)
print(titanic_df.isnull().sum())Here's a breakdown of what it does:
- It fills missing values in the 'Age' column with the median age of the dataset. This is a common approach for handling missing numerical data.
- For the 'Embarked' column, it fills missing values with the most frequent value (mode) in that column. This is often used for categorical data with missing values.
- The 'Cabin' column is dropped entirely due to having too many missing values. This decision was likely made because the high proportion of missing data in this column could potentially introduce more bias if imputed.
- Finally, it prints the sum of null values in each column after these operations. This helps verify that the missing value handling was successful.
This approach to handling missing data is part of the feature engineering process, aiming to prepare the dataset for machine learning algorithms while preserving as much useful information as possible.
6.1.3 Feature Encoding
The Titanic dataset encompasses several categorical variables, notably Sex and Embarked, which require transformation into a numerical format to be compatible with machine learning algorithms. This conversion process is crucial as most machine learning models are designed to work with numerical inputs. To achieve this transformation, we will employ various encoding techniques, with a particular focus on one-hot encoding.
One-hot encoding is a method that creates binary columns for each category within a categorical variable. For instance, the 'Sex' variable would be split into two columns: 'Sex_male' and 'Sex_female', where each passenger would have a '1' in one column and a '0' in the other. This approach allows us to represent categorical data numerically without implying any ordinal relationship between categories.
Additionally, we may consider other encoding techniques such as label encoding for ordinal variables or target encoding for high-cardinality categorical variables, depending on the specific characteristics of each feature. The choice of encoding method can significantly impact model performance, making it a critical step in our feature engineering process.
# One-hot encode the 'Sex' and 'Embarked' columns
titanic_df = pd.get_dummies(titanic_df, columns=['Sex', 'Embarked'], drop_first=True)
print(titanic_df.head())Here's a breakdown of what it does:
- It uses the
pd.get_dummies()function to one-hot encode the 'Sex' and 'Embarked' columns. - The
columns=['Sex', 'Embarked']parameter specifies which columns to encode. - The
drop_first=Trueargument is used to avoid multicollinearity by dropping one of the created columns for each original categorical variable. - The result is stored back in the
titanic_dfDataFrame, effectively replacing the original 'Sex' and 'Embarked' columns with their one-hot encoded versions. - Finally, it prints the first few rows of the updated DataFrame to show the results of the encoding.
This step is crucial in the feature engineering process as it transforms categorical data into a format that can be easily utilized by machine learning algorithms, which typically require numerical inputs.
6.1.4 Feature Scaling
Feature scaling is a crucial step in our feature engineering process, addressing the significant scale disparities among certain features like Fare and Age. These disparities can have detrimental effects on model performance, particularly for algorithms that are sensitive to feature scales, such as logistic regression or K-nearest neighbors. To mitigate these issues and ensure optimal model performance, we will employ standard scaling as our normalization technique.
Standard scaling, also known as z-score normalization, transforms features to have a mean of 0 and a standard deviation of 1. This transformation preserves the shape of the original distribution while bringing all features to a comparable scale. By applying standard scaling to our dataset, we create a level playing field for all features, allowing algorithms to treat them equally and preventing features with larger magnitudes from dominating the learning process.
The benefits of this scaling approach extend beyond just improving model performance. It also enhances the interpretability of our model coefficients, facilitates faster convergence during the training process, and helps in comparing the relative importance of different features. As we proceed with our analysis, this scaling step will prove instrumental in extracting meaningful insights and building robust predictive models.
from sklearn.preprocessing import StandardScaler
scaling_features = ['Age', 'Fare']
scaler = StandardScaler()
titanic_df[scaling_features] = scaler.fit_transform(titanic_df[scaling_features])
print(titanic_df[scaling_features].head())Here's a breakdown of what it does:
- It imports the StandardScaler class from sklearn.preprocessing.
- It defines a list called 'scaling_features' containing 'Age' and 'Fare', which are the features to be scaled.
- It creates an instance of StandardScaler called 'scaler'.
- It applies the fit_transform method of the scaler to the specified features in the titanic_df DataFrame. This step both fits the scaler to the data and transforms it.
- Finally, it prints the head of the scaled features to show the result.
StandardScaler transforms the features to have a mean of 0 and a standard deviation of 1. This is important for many machine learning algorithms that are sensitive to the scale of input features, as it helps to prevent features with larger magnitudes from dominating the model training process.
6.1.5 Feature Creation
Creating new features is a powerful technique that can significantly enhance a model's ability to capture and leverage complex relationships within the data. This process, known as feature engineering, involves deriving new variables from existing ones to provide additional insights or represent the data in a more meaningful way. In this crucial step of our analysis, we will focus on engineering a new feature called FamilySize.
The FamilySize feature will be created by combining two existing variables: SibSp (number of siblings and spouses aboard) and Parch (number of parents and children aboard). By aggregating these related features, we aim to create a more comprehensive representation of a passenger's family unit size. This new feature has the potential to capture important social dynamics and survival patterns that may not be apparent when considering siblings/spouses and parents/children separately.
The rationale behind this feature engineering decision is rooted in the hypothesis that family size could have played a significant role in survival outcomes during the Titanic disaster. For instance, larger families might have faced different challenges or received different treatment compared to individuals traveling alone or in smaller groups. By creating the FamilySize feature, we provide our model with a more nuanced understanding of each passenger's familial context, potentially improving its predictive capabilities.
# Create a new feature 'FamilySize'
titanic_df['FamilySize'] = titanic_df['SibSp'] + titanic_df['Parch'] + 1
# Create a new feature 'IsAlone'
titanic_df['IsAlone'] = (titanic_df['FamilySize'] == 1).astype(int)
print(titanic_df[['SibSp', 'Parch', 'FamilySize', 'IsAlone']].head())This code snippet demonstrates the creation of two new features in the Titanic dataset through feature engineering:
- FamilySize: This feature is created by summing the values of 'SibSp' (number of siblings and spouses aboard), 'Parch' (number of parents and children aboard), and adding 1 (to include the passenger themselves). This provides a comprehensive measure of the total family size for each passenger.
- IsAlone: This is a binary feature that indicates whether a passenger is traveling alone or not. It's derived from the 'FamilySize' feature, where a value of 1 indicates the passenger is alone, and 0 indicates they are with family.
The code then prints the first few rows of the DataFrame, showing these new features alongside the original 'SibSp' and 'Parch' columns for comparison.
These new features aim to capture more nuanced information about each passenger's family context, which could potentially improve the predictive power of the machine learning model for survival prediction.
6.1.6 Feature Selection
Feature selection is a crucial step in the machine learning pipeline that involves identifying and selecting the most relevant features from the dataset. This process helps to reduce dimensionality, improve model performance, and enhance interpretability. In our Titanic survival prediction project, we'll employ feature selection techniques to identify the most informative features for our predictive model.
There are several methods for feature selection, including filter methods (e.g., correlation-based selection), wrapper methods (e.g., recursive feature elimination), and embedded methods (e.g., L1 regularization). For this project, we'll use a filter method called SelectKBest, which selects features based on their statistical relationship with the target variable.
By applying feature selection, we aim to:
- Reduce overfitting by removing irrelevant or redundant features
- Improve model accuracy by focusing on the most predictive features
- Decrease training time by reducing the dataset's dimensionality
- Enhance model interpretability by identifying the most important features
Let's proceed with implementing the SelectKBest method to choose the top features for our Titanic survival prediction model.
from sklearn.feature_selection import SelectKBest, f_classif
X = titanic_df.drop(columns=['Survived', 'PassengerId', 'Name', 'Ticket'])
y = titanic_df['Survived']
# Select top 10 features
selector = SelectKBest(score_func=f_classif, k=10)
X_selected = selector.fit_transform(X, y)
# Get selected feature names
selected_features = X.columns[selector.get_support()].tolist()
print("Selected features:", selected_features)Here's a breakdown of what the code does:
- It imports the necessary functions from scikit-learn's feature_selection module.
- It prepares the feature matrix X by dropping columns that are not needed for prediction ('Survived', 'PassengerId', 'Name', 'Ticket') from the titanic_df DataFrame.
- It defines the target variable y as the 'Survived' column.
- It creates a SelectKBest object with f_classif as the scoring function and k=10, meaning it will select the top 10 features.
- It applies the feature selection to the data using fit_transform(), which both fits the selector to the data and transforms the data to include only the selected features.
- Finally, it retrieves the names of the selected features and prints them.
This feature selection step is crucial in the machine learning pipeline as it helps to identify the most relevant features for predicting survival on the Titanic. By reducing the number of features to the most informative ones, it can improve model performance, reduce overfitting, and enhance interpretability.
6.1.7 Handle Imbalanced Data
In many real-world datasets, including the Titanic dataset, class imbalance is a common issue. This occurs when one class (in our case, survivors or non-survivors) significantly outnumbers the other. Such imbalance can lead to biased models that perform poorly on the minority class.
To address this issue, we'll use a technique called Synthetic Minority Over-sampling Technique (SMOTE). SMOTE works by creating synthetic examples of the minority class, effectively balancing the dataset. This approach can help improve the model's ability to predict both classes accurately.
from imblearn.over_sampling import SMOTE
# Check class distribution
print("Original class distribution:", y.value_counts())
# Apply SMOTE
smote = SMOTE(random_state=42)
X_resampled, y_resampled = smote.fit_resample(X_selected, y)
print("Resampled class distribution:", pd.Series(y_resampled).value_counts())Here's a breakdown of what the code does:
- It imports the SMOTE class from the imblearn.over_sampling module.
- It prints the original class distribution using y.value_counts() to show the imbalance in the dataset.
- It creates a SMOTE object with a random state of 42 for reproducibility.
- It applies SMOTE to the selected features (X_selected) and the target variable (y) using the fit_resample() method. This creates synthetic examples of the minority class to balance the dataset.
- Finally, it prints the resampled class distribution to show how SMOTE has balanced the classes.
This step is crucial in addressing the class imbalance issue, which can lead to biased models. By creating synthetic examples of the minority class, SMOTE helps improve the model's ability to predict both classes accurately.
6.1.8 Model Building and Evaluation
In this crucial phase of our project, we will construct and assess various machine learning models using the engineered features we've developed. This step is essential for determining the effectiveness of our feature engineering efforts and identifying the most suitable model for predicting Titanic survival.
We'll employ multiple algorithms, including Logistic Regression, Random Forest, and Support Vector Machines (SVM). By comparing their performance, we can gain insights into which model best captures the patterns in our engineered dataset. We'll use cross-validation to ensure robust evaluation and metrics such as accuracy, confusion matrix, and classification report to comprehensively assess each model's performance.
This section will demonstrate how our feature engineering work translates into predictive power, highlighting the importance of the entire process in developing effective machine learning solutions.
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC
from sklearn.metrics import accuracy_score, confusion_matrix, classification_report
# Split the data
X_train, X_test, y_train, y_test = train_test_split(X_resampled, y_resampled, test_size=0.2, random_state=42)
# Initialize models
models = {
'Logistic Regression': LogisticRegression(),
'Random Forest': RandomForestClassifier(),
'SVM': SVC()
}
# Train and evaluate models
for name, model in models.items():
# Cross-validation
cv_scores = cross_val_score(model, X_train, y_train, cv=5)
print(f"{name} CV Score: {cv_scores.mean():.4f} (+/- {cv_scores.std() * 2:.4f})")
# Train the model
model.fit(X_train, y_train)
# Make predictions
y_pred = model.predict(X_test)
# Evaluate the model
print(f"{name} Accuracy: {accuracy_score(y_test, y_pred):.4f}")
print(f"{name} Confusion Matrix:\n", confusion_matrix(y_test, y_pred))
print(f"{name} Classification Report:\n", classification_report(y_test, y_pred))
print("\n")Here's a breakdown of what the code does:
- It imports necessary libraries and functions for model training, evaluation, and cross-validation.
- The data is split into training and testing sets using train_test_split.
- Three different models are initialized: Logistic Regression, Random Forest, and Support Vector Machine (SVM).
- For each model, the code performs the following steps:
- Conducts cross-validation using cross_val_score to assess the model's performance across different subsets of the training data.
- Trains the model on the full training set.
- Makes predictions on the test set.
- Evaluates the model's performance using various metrics:
- Accuracy score
- Confusion matrix
- Classification report (which includes precision, recall, and F1-score)
This comprehensive evaluation allows for a comparison of different models' performance on the engineered features, helping to identify which model best captures the patterns in the dataset. The use of cross-validation ensures a robust evaluation by testing the models on different subsets of the data.
6.1.9 Hyperparameter Tuning
Hyperparameter tuning is a crucial step in optimizing machine learning models. It involves finding the best combination of hyperparameters that yield the highest model performance. In this section, we'll use GridSearchCV to systematically search through a predefined set of hyperparameters for our Random Forest model.
Hyperparameters are parameters that are not learned from the data but are set prior to training. For a Random Forest, these may include the number of trees (n_estimators), the maximum depth of trees (max_depth), and the minimum number of samples required to split an internal node (min_samples_split).
By tuning these hyperparameters, we can potentially improve our model's performance and generalization capabilities. This process helps us find the optimal balance between model complexity and performance, reducing the risk of overfitting or underfitting.
from sklearn.model_selection import GridSearchCV
# Example for Random Forest
param_grid = {
'n_estimators': [100, 200, 300],
'max_depth': [5, 10, None],
'min_samples_split': [2, 5, 10]
}
rf = RandomForestClassifier(random_state=42)
grid_search = GridSearchCV(estimator=rf, param_grid=param_grid, cv=5)
grid_search.fit(X_train, y_train)
print("Best parameters:", grid_search.best_params_)
print("Best cross-validation score:", grid_search.best_score_)
# Evaluate the best model
best_model = grid_search.best_estimator_
y_pred = best_model.predict(X_test)
print("Best Model Accuracy:", accuracy_score(y_test, y_pred))Here's a breakdown of what the code does:
- It imports GridSearchCV from scikit-learn, which is used for searching the best parameters for a model.
- A parameter grid (param_grid) is defined with different values for 'n_estimators', 'max_depth', and 'min_samples_split'. These are the hyperparameters we want to optimize.
- A RandomForestClassifier is initialized with a fixed random state for reproducibility.
- GridSearchCV is set up with the Random Forest model, the parameter grid, and 5-fold cross-validation.
- The grid search is performed using fit() on the training data.
- The best parameters and the best cross-validation score are printed.
- Finally, the best model (with optimized parameters) is used to make predictions on the test set, and its accuracy is printed.
This process helps find the optimal combination of hyperparameters that yields the best model performance, potentially improving the model's accuracy and generalization capabilities.
6.1.10 Feature Importance Analysis
Feature importance analysis is a crucial step in understanding which features contribute most significantly to our model's predictions. This analysis helps us identify the most influential factors in determining Titanic passenger survival, providing valuable insights into the dataset and our model's decision-making process.
By examining feature importance, we can:
- Gain a deeper understanding of the factors that most affected survival rates
- Validate our feature engineering efforts by seeing which engineered features are most impactful
- Potentially simplify our model by focusing on the most important features
- Inform future data collection efforts by highlighting the most critical information
In the following code, we'll use our best Random Forest model to calculate and visualize feature importance, providing a clear picture of which features are driving our predictions.
# Using the best Random Forest model
feature_importance = best_model.feature_importances_
feature_names = X.columns[selector.get_support()].tolist()
# Sort features by importance
feature_importance_sorted = sorted(zip(feature_importance, feature_names), reverse=True)
# Plot feature importance
plt.figure(figsize=(10, 6))
plt.bar([x[1] for x in feature_importance_sorted], [x[0] for x in feature_importance_sorted])
plt.title('Feature Importance')
plt.xlabel('Features')
plt.ylabel('Importance')
plt.xticks(rotation=45)
plt.tight_layout()
plt.show()Here's a breakdown of what the code does:
- It extracts feature importance scores from the best Random Forest model using
best_model.feature_importances_. - It retrieves the names of the selected features using
X.columns[selector.get_support()].tolist(). - The feature importances and names are combined and sorted in descending order of importance.
- A bar plot is created to visualize the feature importances:
- The plot is set to a size of 10x6 inches.
- Feature names are placed on the x-axis and their importance scores on the y-axis.
- The plot is given a title, x-label, and y-label.
- X-axis labels are rotated 45 degrees for better readability.
This visualization helps identify which features have the most significant impact on the model's predictions, providing insights into the factors that most influence survival predictions in the Titanic dataset.
6.1.11 Error Analysis
Error analysis is a crucial step in understanding where our model is making mistakes and why. This process involves examining the instances where the model's predictions differ from the actual outcomes. By analyzing these misclassifications, we can gain valuable insights into our model's weaknesses and identify potential areas for improvement.
In this section, we'll look at the characteristics of misclassified samples, comparing their features to those of correctly classified instances. This analysis can reveal patterns or specific subgroups where the model struggles, potentially highlighting the need for additional feature engineering, data collection, or model adjustments.
import pandas as pd
# Convert X_test to DataFrame with column names
X_test_df = pd.DataFrame(X_test, columns=selected_features)
# Identify misclassified samples
misclassified = X_test_df[y_test != y_pred].copy()
misclassified['true_label'] = y_test[y_test != y_pred]
misclassified['predicted_label'] = y_pred[y_test != y_pred]
# Display sample misclassified instances
print("Sample of misclassified instances:")
print(misclassified.head())
# Analyze misclassifications
print("\nMisclassification analysis:")
for feature in selected_features:
print(f"\nFeature: {feature}")
print(misclassified.groupby(['true_label', 'predicted_label'])[feature].mean())Here's an explanation of what the code does:
- It identifies misclassified samples by comparing the actual labels (
y_test) with the predicted labels (y_pred). - It creates a new DataFrame called
misclassified, containing only the incorrectly classified instances from the test set. - It adds two new columns to this DataFrame:
'true_label': the actual label fromy_test'predicted_label': the label predicted by the model (y_pred)
- It prints a sample of these misclassified instances using the
head()function. - Then, it performs a detailed analysis of the misclassifications:
- It iterates through each feature in the
misclassifiedDataFrame. - For each feature, it calculates and prints the mean value grouped by
true_labelandpredicted_label. - This helps in understanding patterns in the model’s mistakes.
- It iterates through each feature in the
Why is this useful?
- It allows us to pinpoint specific features where misclassification occurs.
- It helps identify potential biases in the model.
- It can guide feature engineering improvements or hyperparameter tuning to enhance the model’s performance.
6.1.12 Conclusion
In this project, we applied various feature engineering techniques to the Titanic dataset and built multiple predictive models. We expanded on the original project by including data visualization, feature selection, handling imbalanced data, trying multiple models, implementing cross-validation, performing hyperparameter tuning, analyzing feature importance, and conducting error analysis. These additional steps provide a more comprehensive understanding of the dataset and the impact of feature engineering on model performance.
The results demonstrate the importance of feature engineering in improving model accuracy and interpretability. By carefully selecting, transforming, and creating features, we were able to build more robust predictive models. The feature importance analysis and error analysis provide insights into which factors are most crucial for predicting survival and where the model might be falling short.
This project serves as a thorough example of the feature engineering process and its significance in the machine learning pipeline. It showcases how various techniques can be combined to extract meaningful insights from raw data and improve model performance.
6.1 Project 1: Feature Engineering for Predictive Analytics
This project will focus on applying feature engineering techniques to a dataset to improve the performance of a predictive machine learning model. Feature engineering is crucial for making raw data usable for machine learning algorithms by transforming it into meaningful features that improve model performance.
Project Overview
In this project, we will:
- Explore and preprocess the dataset.
- Apply various feature engineering techniques such as handling missing values, encoding categorical variables, scaling features, and creating new features.
- Build a predictive model using the transformed data to demonstrate the impact of feature engineering on model performance.
- Evaluate the performance of the model before and after feature engineering.
We’ll use the Titanic dataset for this project, as it is well-suited for demonstrating various feature engineering techniques. The task is to predict whether a passenger survived the Titanic disaster based on features like age, gender, ticket class, and fare.
6.1.1 Load and Explore the Dataset
We'll begin by loading the Titanic dataset and conducting a comprehensive initial exploration to gain a thorough understanding of its structure and features. This crucial step involves examining the dataset's dimensions, data types, and basic statistical properties. We'll also investigate the presence of missing values and visualize key relationships between variables to lay a solid foundation for our subsequent feature engineering efforts.
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
# Load the Titanic dataset
url = 'https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv'
titanic_df = pd.read_csv(url)
# Display the first few rows and basic information
print(titanic_df.head())
print(titanic_df.info())
print(titanic_df.describe())
# Visualize missing data
plt.figure(figsize=(10, 6))
sns.heatmap(titanic_df.isnull(), cbar=False, cmap='viridis')
plt.title("Missing Values Heatmap")
plt.show()
# Data Visualization
plt.figure(figsize=(12, 5))
plt.subplot(121)
sns.histplot(titanic_df['Age'].dropna(), kde=True)
plt.title('Age Distribution')
plt.subplot(122)
sns.boxplot(x='Pclass', y='Fare', data=titanic_df)
plt.title('Fare Distribution by Passenger Class')
plt.tight_layout()
plt.show()
# Correlation matrix
# Select only numeric columns for correlation
numeric_cols = titanic_df.select_dtypes(include=['number'])
corr_matrix = numeric_cols.corr()
plt.figure(figsize=(10, 8))
sns.heatmap(corr_matrix, annot=True, cmap='coolwarm')
plt.title('Correlation Matrix')
plt.show()Here's a breakdown of what it does:
- Imports necessary libraries: pandas for data manipulation, matplotlib and seaborn for visualization
- Loads the Titanic dataset from a URL using pandas
- Displays basic information about the dataset:
- First few rows (head())
- Dataset info (info())
- Statistical summary (describe())
- Creates visualizations:
- A heatmap to show missing values in the dataset
- A histogram of the Age distribution
- A boxplot showing Fare distribution by Passenger Class
- A correlation matrix heatmap to show relationships between numerical features
This code is part of the data exploration and preprocessing step, which is crucial for understanding the dataset before applying feature engineering techniques. It helps identify missing data, visualize distributions, and understand relationships between variables, laying the groundwork for subsequent analysis and model building.
6.1.2 Handle Missing Data
The Titanic dataset presents several features with missing values, notably including Age and Cabin. Addressing these missing data points is a crucial step in our feature engineering process.
For the Age feature, we'll employ imputation techniques to fill in the gaps with statistically appropriate values, such as the median age or predictions based on other correlated features. In the case of the Cabin feature, given its high proportion of missing entries, we'll carefully evaluate whether to attempt imputation or to exclude it from our analysis.
This decision will be based on the potential information value of the feature versus the risk of introducing bias through imputation. By systematically handling these missing values, we aim to maximize the usable information in our dataset while maintaining the integrity of our subsequent analyses.
# Fill missing values in the 'Age' column with the median age
titanic_df['Age'].fillna(titanic_df['Age'].median(), inplace=True)
# Fill missing values in the 'Embarked' column with the most frequent value
titanic_df['Embarked'].fillna(titanic_df['Embarked'].mode()[0], inplace=True)
# Drop the 'Cabin' column due to too many missing values
titanic_df.drop(columns=['Cabin'], inplace=True)
print(titanic_df.isnull().sum())Here's a breakdown of what it does:
- It fills missing values in the 'Age' column with the median age of the dataset. This is a common approach for handling missing numerical data.
- For the 'Embarked' column, it fills missing values with the most frequent value (mode) in that column. This is often used for categorical data with missing values.
- The 'Cabin' column is dropped entirely due to having too many missing values. This decision was likely made because the high proportion of missing data in this column could potentially introduce more bias if imputed.
- Finally, it prints the sum of null values in each column after these operations. This helps verify that the missing value handling was successful.
This approach to handling missing data is part of the feature engineering process, aiming to prepare the dataset for machine learning algorithms while preserving as much useful information as possible.
6.1.3 Feature Encoding
The Titanic dataset encompasses several categorical variables, notably Sex and Embarked, which require transformation into a numerical format to be compatible with machine learning algorithms. This conversion process is crucial as most machine learning models are designed to work with numerical inputs. To achieve this transformation, we will employ various encoding techniques, with a particular focus on one-hot encoding.
One-hot encoding is a method that creates binary columns for each category within a categorical variable. For instance, the 'Sex' variable would be split into two columns: 'Sex_male' and 'Sex_female', where each passenger would have a '1' in one column and a '0' in the other. This approach allows us to represent categorical data numerically without implying any ordinal relationship between categories.
Additionally, we may consider other encoding techniques such as label encoding for ordinal variables or target encoding for high-cardinality categorical variables, depending on the specific characteristics of each feature. The choice of encoding method can significantly impact model performance, making it a critical step in our feature engineering process.
# One-hot encode the 'Sex' and 'Embarked' columns
titanic_df = pd.get_dummies(titanic_df, columns=['Sex', 'Embarked'], drop_first=True)
print(titanic_df.head())Here's a breakdown of what it does:
- It uses the
pd.get_dummies()function to one-hot encode the 'Sex' and 'Embarked' columns. - The
columns=['Sex', 'Embarked']parameter specifies which columns to encode. - The
drop_first=Trueargument is used to avoid multicollinearity by dropping one of the created columns for each original categorical variable. - The result is stored back in the
titanic_dfDataFrame, effectively replacing the original 'Sex' and 'Embarked' columns with their one-hot encoded versions. - Finally, it prints the first few rows of the updated DataFrame to show the results of the encoding.
This step is crucial in the feature engineering process as it transforms categorical data into a format that can be easily utilized by machine learning algorithms, which typically require numerical inputs.
6.1.4 Feature Scaling
Feature scaling is a crucial step in our feature engineering process, addressing the significant scale disparities among certain features like Fare and Age. These disparities can have detrimental effects on model performance, particularly for algorithms that are sensitive to feature scales, such as logistic regression or K-nearest neighbors. To mitigate these issues and ensure optimal model performance, we will employ standard scaling as our normalization technique.
Standard scaling, also known as z-score normalization, transforms features to have a mean of 0 and a standard deviation of 1. This transformation preserves the shape of the original distribution while bringing all features to a comparable scale. By applying standard scaling to our dataset, we create a level playing field for all features, allowing algorithms to treat them equally and preventing features with larger magnitudes from dominating the learning process.
The benefits of this scaling approach extend beyond just improving model performance. It also enhances the interpretability of our model coefficients, facilitates faster convergence during the training process, and helps in comparing the relative importance of different features. As we proceed with our analysis, this scaling step will prove instrumental in extracting meaningful insights and building robust predictive models.
from sklearn.preprocessing import StandardScaler
scaling_features = ['Age', 'Fare']
scaler = StandardScaler()
titanic_df[scaling_features] = scaler.fit_transform(titanic_df[scaling_features])
print(titanic_df[scaling_features].head())Here's a breakdown of what it does:
- It imports the StandardScaler class from sklearn.preprocessing.
- It defines a list called 'scaling_features' containing 'Age' and 'Fare', which are the features to be scaled.
- It creates an instance of StandardScaler called 'scaler'.
- It applies the fit_transform method of the scaler to the specified features in the titanic_df DataFrame. This step both fits the scaler to the data and transforms it.
- Finally, it prints the head of the scaled features to show the result.
StandardScaler transforms the features to have a mean of 0 and a standard deviation of 1. This is important for many machine learning algorithms that are sensitive to the scale of input features, as it helps to prevent features with larger magnitudes from dominating the model training process.
6.1.5 Feature Creation
Creating new features is a powerful technique that can significantly enhance a model's ability to capture and leverage complex relationships within the data. This process, known as feature engineering, involves deriving new variables from existing ones to provide additional insights or represent the data in a more meaningful way. In this crucial step of our analysis, we will focus on engineering a new feature called FamilySize.
The FamilySize feature will be created by combining two existing variables: SibSp (number of siblings and spouses aboard) and Parch (number of parents and children aboard). By aggregating these related features, we aim to create a more comprehensive representation of a passenger's family unit size. This new feature has the potential to capture important social dynamics and survival patterns that may not be apparent when considering siblings/spouses and parents/children separately.
The rationale behind this feature engineering decision is rooted in the hypothesis that family size could have played a significant role in survival outcomes during the Titanic disaster. For instance, larger families might have faced different challenges or received different treatment compared to individuals traveling alone or in smaller groups. By creating the FamilySize feature, we provide our model with a more nuanced understanding of each passenger's familial context, potentially improving its predictive capabilities.
# Create a new feature 'FamilySize'
titanic_df['FamilySize'] = titanic_df['SibSp'] + titanic_df['Parch'] + 1
# Create a new feature 'IsAlone'
titanic_df['IsAlone'] = (titanic_df['FamilySize'] == 1).astype(int)
print(titanic_df[['SibSp', 'Parch', 'FamilySize', 'IsAlone']].head())This code snippet demonstrates the creation of two new features in the Titanic dataset through feature engineering:
- FamilySize: This feature is created by summing the values of 'SibSp' (number of siblings and spouses aboard), 'Parch' (number of parents and children aboard), and adding 1 (to include the passenger themselves). This provides a comprehensive measure of the total family size for each passenger.
- IsAlone: This is a binary feature that indicates whether a passenger is traveling alone or not. It's derived from the 'FamilySize' feature, where a value of 1 indicates the passenger is alone, and 0 indicates they are with family.
The code then prints the first few rows of the DataFrame, showing these new features alongside the original 'SibSp' and 'Parch' columns for comparison.
These new features aim to capture more nuanced information about each passenger's family context, which could potentially improve the predictive power of the machine learning model for survival prediction.
6.1.6 Feature Selection
Feature selection is a crucial step in the machine learning pipeline that involves identifying and selecting the most relevant features from the dataset. This process helps to reduce dimensionality, improve model performance, and enhance interpretability. In our Titanic survival prediction project, we'll employ feature selection techniques to identify the most informative features for our predictive model.
There are several methods for feature selection, including filter methods (e.g., correlation-based selection), wrapper methods (e.g., recursive feature elimination), and embedded methods (e.g., L1 regularization). For this project, we'll use a filter method called SelectKBest, which selects features based on their statistical relationship with the target variable.
By applying feature selection, we aim to:
- Reduce overfitting by removing irrelevant or redundant features
- Improve model accuracy by focusing on the most predictive features
- Decrease training time by reducing the dataset's dimensionality
- Enhance model interpretability by identifying the most important features
Let's proceed with implementing the SelectKBest method to choose the top features for our Titanic survival prediction model.
from sklearn.feature_selection import SelectKBest, f_classif
X = titanic_df.drop(columns=['Survived', 'PassengerId', 'Name', 'Ticket'])
y = titanic_df['Survived']
# Select top 10 features
selector = SelectKBest(score_func=f_classif, k=10)
X_selected = selector.fit_transform(X, y)
# Get selected feature names
selected_features = X.columns[selector.get_support()].tolist()
print("Selected features:", selected_features)Here's a breakdown of what the code does:
- It imports the necessary functions from scikit-learn's feature_selection module.
- It prepares the feature matrix X by dropping columns that are not needed for prediction ('Survived', 'PassengerId', 'Name', 'Ticket') from the titanic_df DataFrame.
- It defines the target variable y as the 'Survived' column.
- It creates a SelectKBest object with f_classif as the scoring function and k=10, meaning it will select the top 10 features.
- It applies the feature selection to the data using fit_transform(), which both fits the selector to the data and transforms the data to include only the selected features.
- Finally, it retrieves the names of the selected features and prints them.
This feature selection step is crucial in the machine learning pipeline as it helps to identify the most relevant features for predicting survival on the Titanic. By reducing the number of features to the most informative ones, it can improve model performance, reduce overfitting, and enhance interpretability.
6.1.7 Handle Imbalanced Data
In many real-world datasets, including the Titanic dataset, class imbalance is a common issue. This occurs when one class (in our case, survivors or non-survivors) significantly outnumbers the other. Such imbalance can lead to biased models that perform poorly on the minority class.
To address this issue, we'll use a technique called Synthetic Minority Over-sampling Technique (SMOTE). SMOTE works by creating synthetic examples of the minority class, effectively balancing the dataset. This approach can help improve the model's ability to predict both classes accurately.
from imblearn.over_sampling import SMOTE
# Check class distribution
print("Original class distribution:", y.value_counts())
# Apply SMOTE
smote = SMOTE(random_state=42)
X_resampled, y_resampled = smote.fit_resample(X_selected, y)
print("Resampled class distribution:", pd.Series(y_resampled).value_counts())Here's a breakdown of what the code does:
- It imports the SMOTE class from the imblearn.over_sampling module.
- It prints the original class distribution using y.value_counts() to show the imbalance in the dataset.
- It creates a SMOTE object with a random state of 42 for reproducibility.
- It applies SMOTE to the selected features (X_selected) and the target variable (y) using the fit_resample() method. This creates synthetic examples of the minority class to balance the dataset.
- Finally, it prints the resampled class distribution to show how SMOTE has balanced the classes.
This step is crucial in addressing the class imbalance issue, which can lead to biased models. By creating synthetic examples of the minority class, SMOTE helps improve the model's ability to predict both classes accurately.
6.1.8 Model Building and Evaluation
In this crucial phase of our project, we will construct and assess various machine learning models using the engineered features we've developed. This step is essential for determining the effectiveness of our feature engineering efforts and identifying the most suitable model for predicting Titanic survival.
We'll employ multiple algorithms, including Logistic Regression, Random Forest, and Support Vector Machines (SVM). By comparing their performance, we can gain insights into which model best captures the patterns in our engineered dataset. We'll use cross-validation to ensure robust evaluation and metrics such as accuracy, confusion matrix, and classification report to comprehensively assess each model's performance.
This section will demonstrate how our feature engineering work translates into predictive power, highlighting the importance of the entire process in developing effective machine learning solutions.
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC
from sklearn.metrics import accuracy_score, confusion_matrix, classification_report
# Split the data
X_train, X_test, y_train, y_test = train_test_split(X_resampled, y_resampled, test_size=0.2, random_state=42)
# Initialize models
models = {
'Logistic Regression': LogisticRegression(),
'Random Forest': RandomForestClassifier(),
'SVM': SVC()
}
# Train and evaluate models
for name, model in models.items():
# Cross-validation
cv_scores = cross_val_score(model, X_train, y_train, cv=5)
print(f"{name} CV Score: {cv_scores.mean():.4f} (+/- {cv_scores.std() * 2:.4f})")
# Train the model
model.fit(X_train, y_train)
# Make predictions
y_pred = model.predict(X_test)
# Evaluate the model
print(f"{name} Accuracy: {accuracy_score(y_test, y_pred):.4f}")
print(f"{name} Confusion Matrix:\n", confusion_matrix(y_test, y_pred))
print(f"{name} Classification Report:\n", classification_report(y_test, y_pred))
print("\n")Here's a breakdown of what the code does:
- It imports necessary libraries and functions for model training, evaluation, and cross-validation.
- The data is split into training and testing sets using train_test_split.
- Three different models are initialized: Logistic Regression, Random Forest, and Support Vector Machine (SVM).
- For each model, the code performs the following steps:
- Conducts cross-validation using cross_val_score to assess the model's performance across different subsets of the training data.
- Trains the model on the full training set.
- Makes predictions on the test set.
- Evaluates the model's performance using various metrics:
- Accuracy score
- Confusion matrix
- Classification report (which includes precision, recall, and F1-score)
This comprehensive evaluation allows for a comparison of different models' performance on the engineered features, helping to identify which model best captures the patterns in the dataset. The use of cross-validation ensures a robust evaluation by testing the models on different subsets of the data.
6.1.9 Hyperparameter Tuning
Hyperparameter tuning is a crucial step in optimizing machine learning models. It involves finding the best combination of hyperparameters that yield the highest model performance. In this section, we'll use GridSearchCV to systematically search through a predefined set of hyperparameters for our Random Forest model.
Hyperparameters are parameters that are not learned from the data but are set prior to training. For a Random Forest, these may include the number of trees (n_estimators), the maximum depth of trees (max_depth), and the minimum number of samples required to split an internal node (min_samples_split).
By tuning these hyperparameters, we can potentially improve our model's performance and generalization capabilities. This process helps us find the optimal balance between model complexity and performance, reducing the risk of overfitting or underfitting.
from sklearn.model_selection import GridSearchCV
# Example for Random Forest
param_grid = {
'n_estimators': [100, 200, 300],
'max_depth': [5, 10, None],
'min_samples_split': [2, 5, 10]
}
rf = RandomForestClassifier(random_state=42)
grid_search = GridSearchCV(estimator=rf, param_grid=param_grid, cv=5)
grid_search.fit(X_train, y_train)
print("Best parameters:", grid_search.best_params_)
print("Best cross-validation score:", grid_search.best_score_)
# Evaluate the best model
best_model = grid_search.best_estimator_
y_pred = best_model.predict(X_test)
print("Best Model Accuracy:", accuracy_score(y_test, y_pred))Here's a breakdown of what the code does:
- It imports GridSearchCV from scikit-learn, which is used for searching the best parameters for a model.
- A parameter grid (param_grid) is defined with different values for 'n_estimators', 'max_depth', and 'min_samples_split'. These are the hyperparameters we want to optimize.
- A RandomForestClassifier is initialized with a fixed random state for reproducibility.
- GridSearchCV is set up with the Random Forest model, the parameter grid, and 5-fold cross-validation.
- The grid search is performed using fit() on the training data.
- The best parameters and the best cross-validation score are printed.
- Finally, the best model (with optimized parameters) is used to make predictions on the test set, and its accuracy is printed.
This process helps find the optimal combination of hyperparameters that yields the best model performance, potentially improving the model's accuracy and generalization capabilities.
6.1.10 Feature Importance Analysis
Feature importance analysis is a crucial step in understanding which features contribute most significantly to our model's predictions. This analysis helps us identify the most influential factors in determining Titanic passenger survival, providing valuable insights into the dataset and our model's decision-making process.
By examining feature importance, we can:
- Gain a deeper understanding of the factors that most affected survival rates
- Validate our feature engineering efforts by seeing which engineered features are most impactful
- Potentially simplify our model by focusing on the most important features
- Inform future data collection efforts by highlighting the most critical information
In the following code, we'll use our best Random Forest model to calculate and visualize feature importance, providing a clear picture of which features are driving our predictions.
# Using the best Random Forest model
feature_importance = best_model.feature_importances_
feature_names = X.columns[selector.get_support()].tolist()
# Sort features by importance
feature_importance_sorted = sorted(zip(feature_importance, feature_names), reverse=True)
# Plot feature importance
plt.figure(figsize=(10, 6))
plt.bar([x[1] for x in feature_importance_sorted], [x[0] for x in feature_importance_sorted])
plt.title('Feature Importance')
plt.xlabel('Features')
plt.ylabel('Importance')
plt.xticks(rotation=45)
plt.tight_layout()
plt.show()Here's a breakdown of what the code does:
- It extracts feature importance scores from the best Random Forest model using
best_model.feature_importances_. - It retrieves the names of the selected features using
X.columns[selector.get_support()].tolist(). - The feature importances and names are combined and sorted in descending order of importance.
- A bar plot is created to visualize the feature importances:
- The plot is set to a size of 10x6 inches.
- Feature names are placed on the x-axis and their importance scores on the y-axis.
- The plot is given a title, x-label, and y-label.
- X-axis labels are rotated 45 degrees for better readability.
This visualization helps identify which features have the most significant impact on the model's predictions, providing insights into the factors that most influence survival predictions in the Titanic dataset.
6.1.11 Error Analysis
Error analysis is a crucial step in understanding where our model is making mistakes and why. This process involves examining the instances where the model's predictions differ from the actual outcomes. By analyzing these misclassifications, we can gain valuable insights into our model's weaknesses and identify potential areas for improvement.
In this section, we'll look at the characteristics of misclassified samples, comparing their features to those of correctly classified instances. This analysis can reveal patterns or specific subgroups where the model struggles, potentially highlighting the need for additional feature engineering, data collection, or model adjustments.
import pandas as pd
# Convert X_test to DataFrame with column names
X_test_df = pd.DataFrame(X_test, columns=selected_features)
# Identify misclassified samples
misclassified = X_test_df[y_test != y_pred].copy()
misclassified['true_label'] = y_test[y_test != y_pred]
misclassified['predicted_label'] = y_pred[y_test != y_pred]
# Display sample misclassified instances
print("Sample of misclassified instances:")
print(misclassified.head())
# Analyze misclassifications
print("\nMisclassification analysis:")
for feature in selected_features:
print(f"\nFeature: {feature}")
print(misclassified.groupby(['true_label', 'predicted_label'])[feature].mean())Here's an explanation of what the code does:
- It identifies misclassified samples by comparing the actual labels (
y_test) with the predicted labels (y_pred). - It creates a new DataFrame called
misclassified, containing only the incorrectly classified instances from the test set. - It adds two new columns to this DataFrame:
'true_label': the actual label fromy_test'predicted_label': the label predicted by the model (y_pred)
- It prints a sample of these misclassified instances using the
head()function. - Then, it performs a detailed analysis of the misclassifications:
- It iterates through each feature in the
misclassifiedDataFrame. - For each feature, it calculates and prints the mean value grouped by
true_labelandpredicted_label. - This helps in understanding patterns in the model’s mistakes.
- It iterates through each feature in the
Why is this useful?
- It allows us to pinpoint specific features where misclassification occurs.
- It helps identify potential biases in the model.
- It can guide feature engineering improvements or hyperparameter tuning to enhance the model’s performance.
6.1.12 Conclusion
In this project, we applied various feature engineering techniques to the Titanic dataset and built multiple predictive models. We expanded on the original project by including data visualization, feature selection, handling imbalanced data, trying multiple models, implementing cross-validation, performing hyperparameter tuning, analyzing feature importance, and conducting error analysis. These additional steps provide a more comprehensive understanding of the dataset and the impact of feature engineering on model performance.
The results demonstrate the importance of feature engineering in improving model accuracy and interpretability. By carefully selecting, transforming, and creating features, we were able to build more robust predictive models. The feature importance analysis and error analysis provide insights into which factors are most crucial for predicting survival and where the model might be falling short.
This project serves as a thorough example of the feature engineering process and its significance in the machine learning pipeline. It showcases how various techniques can be combined to extract meaningful insights from raw data and improve model performance.