Code icon

The App is Under a Quick Maintenance

We apologize for the inconvenience. Please come back later

Menu iconMenu iconGenerative Deep Learning with Python
Generative Deep Learning with Python

Chapter 2: Understanding Generative Models

2.5 Practical Exercises of Chapter 2: Understanding Generative Models

The exercises in this section will provide you with ample opportunity to apply the concepts that you've learned in this chapter. You will have the chance to delve more deeply into the various types of generative models that you've been introduced to, explore their loss functions in greater detail, and work through some of the challenges that arise when training these models. 

Specifically, you will be able to experiment with different techniques for fine-tuning your models, explore the use of regularization to prevent overfitting, and consider the various trade-offs that arise when selecting different hyperparameters.

The exercises will help you to develop a deeper understanding of how to apply the techniques that you've learned in a practical setting, by providing you with hands-on experience in working with real-world data. The skills that you develop through this section will help you to become a more effective practitioner in the field of machine learning, and to take on more advanced challenges as you continue to progress in your studies.

Exercise 2.5.1: Implementing a Variational Autoencoder (VAE)

In this exercise, you will implement a basic VAE in TensorFlow. The aim is to create a VAE that can generate new samples from the MNIST dataset.

Example:

Here's an example of how to implement a simple VAE in TensorFlow:

import tensorflow as tf
from tensorflow.keras import layers

class VAE(tf.keras.Model):
    def __init__(self, latent_dim):
        super(VAE, self).__init__()
        self.latent_dim = latent_dim
        self.encoder = tf.keras.Sequential([
            layers.InputLayer(input_shape=(28, 28, 1)),
            layers.Conv2D(filters=32, kernel_size=3, strides=(2, 2), activation='relu'),
            layers.Conv2D(filters=64, kernel_size=3, strides=(2, 2), activation='relu'),
            layers.Flatten(),
            layers.Dense(latent_dim + latent_dim),
        ])
        self.decoder = tf.keras.Sequential([
            layers.InputLayer(input_shape=(latent_dim,)),
            layers.Dense(units=7*7*32, activation=tf.nn.relu),
            layers.Reshape(target_shape=(7, 7, 32)),
            layers.Conv2DTranspose(filters=64, kernel_size=3, strides=2, padding='same', activation='relu'),
            layers.Conv2DTranspose(filters=32, kernel_size=3, strides=2, padding='same', activation='relu'),
            layers.Conv2DTranspose(filters=1, kernel_size=3, strides=1, padding='same'),
        ])

    def encode(self, x):
        mean, logvar = tf.split(self.encoder(x), num_or_size_splits=2, axis=1)
        return mean, logvar

    def reparameterize(self, mean, logvar):
        epsilon = tf.random.normal(shape=mean.shape)
        return mean + tf.exp(0.5 * logvar) * epsilon

    def decode(self, z, apply_sigmoid=False):
        logits = self.decoder(z)
        if apply_sigmoid:
            probs = tf.sigmoid(logits)
            return probs
        return logits

    def call(self, inputs):
        mean, logvar = self.encode(inputs)
        z = self.reparameterize(mean, logvar)
        return self.decode(z)

Exercise 2.5.2: Implementing a Generative Adversarial Network (GAN)

Implement a GAN to generate new samples from the MNIST dataset. Pay particular attention to the loss functions of the generator and discriminator.

Example:

Here's an example of how to implement a simple GAN in TensorFlow:

import tensorflow as tf
from tensorflow.keras import layers

def make_generator_model():
    model = tf.keras.Sequential()
    model.add(layers.Dense(7*7*256, use_bias=False, input_shape=(100,)))
    model.add(layers.BatchNormalization())
    model.add(layers.LeakyReLU())

    model.add(layers.Reshape((7, 7, 256)))
    assert model.output_shape == (None, 7, 7, 256)  # Note: None is the batch size

    model.add(layers.Conv2DTranspose(128, (5, 5), strides=(1, 1), padding='same', use_bias=False))
    assert model.output_shape == (None, 7, 7, 128)
    model.add(layers.BatchNormalization())
    model.add(layers.LeakyReLU())

    model.add(layers.Conv2DTranspose(64, (5, 5), strides=(2, 2), padding='same', use_bias=False))
    assert model.output_shape == (None, 14, 14, 64)
    model.add(layers.BatchNormalization())
    model.add(layers.LeakyReLU())

    model.add(layers.Conv2DTranspose(1, (5, 5), strides=(2, 2), padding='same', use_bias=False, activation='tanh'))
    assert model.output_shape == (None, 28, 28, 1)

    return model

def make_discriminator_model():
    model = tf.keras.Sequential()
    model.add(layers.Conv2D(64, (5, 5), strides=(2, 2), padding='same', input_shape=[28, 28, 1]))
    model.add(layers.LeakyReLU())
    model.add(layers.Dropout(0.3))

    model.add(layers.Conv2D(128, (5, 5), strides=(2, 2), padding='same'))
    model.add(layers.LeakyReLU())
    model.add(layers.Dropout(0.3))

    model.add(layers.Flatten())
    model.add(layers.Dense(1))

    return model

Exercise 2.5.3: Experimenting with Loss Functions

In this exercise, you'll modify the GAN you implemented in Exercise 2.5.2 to use a different loss function. Specifically, implement the least squares GAN loss and observe any differences in training and output quality. 

Example:

Here's an example of how to implement the least squares GAN loss:

def generator_loss_LSGAN(fake_output):
    return tf.reduce_mean((fake_output - 1) ** 2)

def discriminator_loss_LSGAN(real_output, fake_output):
    return 0.5 * (tf.reduce_mean((real_output - 1) ** 2) + tf.reduce_mean(fake_output ** 2))

Exercise 2.5.4: Mode Collapse and Potential Solutions

In this exercise, we'll intentionally cause mode collapse in a GAN and then attempt to mitigate it. First, train a GAN on a diverse dataset but restrict the output size so that the GAN cannot reproduce the diversity of the input data. Observe the mode collapse. Then, implement one of the potential solutions to mode collapse discussed in this chapter, such as modifying the loss function or changing the model architecture. 

Remember to experiment and explore different parameters, architectures, and techniques. The more you experiment, the better you'll understand the nuances of training generative models.

In the next chapter, we will go deeper into specific types of generative models and their applications. Stay tuned! 

Chapter 2 Conclusion

In this chapter, we delved into the fascinating world of generative models. We started by understanding the concept of generative models and their importance in the world of deep learning. These models, as we learned, are capable of generating new, unseen data that share the same underlying patterns as the training data. This has far-reaching implications for a variety of fields, including art, entertainment, healthcare, and more.

We then looked at different types of generative models, including Variational Autoencoders (VAEs), Generative Adversarial Networks (GANs), and Autoregressive models. Each type has its unique strengths and characteristics that make them suitable for different types of problems.

Next, we dug into the nuances of training generative models, exploring concepts such as the likelihood, maximum likelihood estimation, and the evidence lower bound (ELBO). We learned how these concepts play a role in training our models and ensuring that they learn a robust representation of the data distribution.

Finally, we addressed some of the challenges that come with training generative models. Issues such as mode collapse and vanishing gradients can hamper the performance of our models, but we also discussed potential solutions to these problems.

The practical exercises at the end of this chapter were designed to help you apply these theoretical concepts. By implementing and experimenting with different types of generative models, you gain hands-on experience and a deeper understanding of the material.

In the next chapter, we'll dive into more details about Generative Adversarial Networks (GANs). We will explore their architecture, how they work, and how to implement them in TensorFlow. Stay tuned for more exciting insights into the world of generative models!

2.5 Practical Exercises of Chapter 2: Understanding Generative Models

The exercises in this section will provide you with ample opportunity to apply the concepts that you've learned in this chapter. You will have the chance to delve more deeply into the various types of generative models that you've been introduced to, explore their loss functions in greater detail, and work through some of the challenges that arise when training these models. 

Specifically, you will be able to experiment with different techniques for fine-tuning your models, explore the use of regularization to prevent overfitting, and consider the various trade-offs that arise when selecting different hyperparameters.

The exercises will help you to develop a deeper understanding of how to apply the techniques that you've learned in a practical setting, by providing you with hands-on experience in working with real-world data. The skills that you develop through this section will help you to become a more effective practitioner in the field of machine learning, and to take on more advanced challenges as you continue to progress in your studies.

Exercise 2.5.1: Implementing a Variational Autoencoder (VAE)

In this exercise, you will implement a basic VAE in TensorFlow. The aim is to create a VAE that can generate new samples from the MNIST dataset.

Example:

Here's an example of how to implement a simple VAE in TensorFlow:

import tensorflow as tf
from tensorflow.keras import layers

class VAE(tf.keras.Model):
    def __init__(self, latent_dim):
        super(VAE, self).__init__()
        self.latent_dim = latent_dim
        self.encoder = tf.keras.Sequential([
            layers.InputLayer(input_shape=(28, 28, 1)),
            layers.Conv2D(filters=32, kernel_size=3, strides=(2, 2), activation='relu'),
            layers.Conv2D(filters=64, kernel_size=3, strides=(2, 2), activation='relu'),
            layers.Flatten(),
            layers.Dense(latent_dim + latent_dim),
        ])
        self.decoder = tf.keras.Sequential([
            layers.InputLayer(input_shape=(latent_dim,)),
            layers.Dense(units=7*7*32, activation=tf.nn.relu),
            layers.Reshape(target_shape=(7, 7, 32)),
            layers.Conv2DTranspose(filters=64, kernel_size=3, strides=2, padding='same', activation='relu'),
            layers.Conv2DTranspose(filters=32, kernel_size=3, strides=2, padding='same', activation='relu'),
            layers.Conv2DTranspose(filters=1, kernel_size=3, strides=1, padding='same'),
        ])

    def encode(self, x):
        mean, logvar = tf.split(self.encoder(x), num_or_size_splits=2, axis=1)
        return mean, logvar

    def reparameterize(self, mean, logvar):
        epsilon = tf.random.normal(shape=mean.shape)
        return mean + tf.exp(0.5 * logvar) * epsilon

    def decode(self, z, apply_sigmoid=False):
        logits = self.decoder(z)
        if apply_sigmoid:
            probs = tf.sigmoid(logits)
            return probs
        return logits

    def call(self, inputs):
        mean, logvar = self.encode(inputs)
        z = self.reparameterize(mean, logvar)
        return self.decode(z)

Exercise 2.5.2: Implementing a Generative Adversarial Network (GAN)

Implement a GAN to generate new samples from the MNIST dataset. Pay particular attention to the loss functions of the generator and discriminator.

Example:

Here's an example of how to implement a simple GAN in TensorFlow:

import tensorflow as tf
from tensorflow.keras import layers

def make_generator_model():
    model = tf.keras.Sequential()
    model.add(layers.Dense(7*7*256, use_bias=False, input_shape=(100,)))
    model.add(layers.BatchNormalization())
    model.add(layers.LeakyReLU())

    model.add(layers.Reshape((7, 7, 256)))
    assert model.output_shape == (None, 7, 7, 256)  # Note: None is the batch size

    model.add(layers.Conv2DTranspose(128, (5, 5), strides=(1, 1), padding='same', use_bias=False))
    assert model.output_shape == (None, 7, 7, 128)
    model.add(layers.BatchNormalization())
    model.add(layers.LeakyReLU())

    model.add(layers.Conv2DTranspose(64, (5, 5), strides=(2, 2), padding='same', use_bias=False))
    assert model.output_shape == (None, 14, 14, 64)
    model.add(layers.BatchNormalization())
    model.add(layers.LeakyReLU())

    model.add(layers.Conv2DTranspose(1, (5, 5), strides=(2, 2), padding='same', use_bias=False, activation='tanh'))
    assert model.output_shape == (None, 28, 28, 1)

    return model

def make_discriminator_model():
    model = tf.keras.Sequential()
    model.add(layers.Conv2D(64, (5, 5), strides=(2, 2), padding='same', input_shape=[28, 28, 1]))
    model.add(layers.LeakyReLU())
    model.add(layers.Dropout(0.3))

    model.add(layers.Conv2D(128, (5, 5), strides=(2, 2), padding='same'))
    model.add(layers.LeakyReLU())
    model.add(layers.Dropout(0.3))

    model.add(layers.Flatten())
    model.add(layers.Dense(1))

    return model

Exercise 2.5.3: Experimenting with Loss Functions

In this exercise, you'll modify the GAN you implemented in Exercise 2.5.2 to use a different loss function. Specifically, implement the least squares GAN loss and observe any differences in training and output quality. 

Example:

Here's an example of how to implement the least squares GAN loss:

def generator_loss_LSGAN(fake_output):
    return tf.reduce_mean((fake_output - 1) ** 2)

def discriminator_loss_LSGAN(real_output, fake_output):
    return 0.5 * (tf.reduce_mean((real_output - 1) ** 2) + tf.reduce_mean(fake_output ** 2))

Exercise 2.5.4: Mode Collapse and Potential Solutions

In this exercise, we'll intentionally cause mode collapse in a GAN and then attempt to mitigate it. First, train a GAN on a diverse dataset but restrict the output size so that the GAN cannot reproduce the diversity of the input data. Observe the mode collapse. Then, implement one of the potential solutions to mode collapse discussed in this chapter, such as modifying the loss function or changing the model architecture. 

Remember to experiment and explore different parameters, architectures, and techniques. The more you experiment, the better you'll understand the nuances of training generative models.

In the next chapter, we will go deeper into specific types of generative models and their applications. Stay tuned! 

Chapter 2 Conclusion

In this chapter, we delved into the fascinating world of generative models. We started by understanding the concept of generative models and their importance in the world of deep learning. These models, as we learned, are capable of generating new, unseen data that share the same underlying patterns as the training data. This has far-reaching implications for a variety of fields, including art, entertainment, healthcare, and more.

We then looked at different types of generative models, including Variational Autoencoders (VAEs), Generative Adversarial Networks (GANs), and Autoregressive models. Each type has its unique strengths and characteristics that make them suitable for different types of problems.

Next, we dug into the nuances of training generative models, exploring concepts such as the likelihood, maximum likelihood estimation, and the evidence lower bound (ELBO). We learned how these concepts play a role in training our models and ensuring that they learn a robust representation of the data distribution.

Finally, we addressed some of the challenges that come with training generative models. Issues such as mode collapse and vanishing gradients can hamper the performance of our models, but we also discussed potential solutions to these problems.

The practical exercises at the end of this chapter were designed to help you apply these theoretical concepts. By implementing and experimenting with different types of generative models, you gain hands-on experience and a deeper understanding of the material.

In the next chapter, we'll dive into more details about Generative Adversarial Networks (GANs). We will explore their architecture, how they work, and how to implement them in TensorFlow. Stay tuned for more exciting insights into the world of generative models!

2.5 Practical Exercises of Chapter 2: Understanding Generative Models

The exercises in this section will provide you with ample opportunity to apply the concepts that you've learned in this chapter. You will have the chance to delve more deeply into the various types of generative models that you've been introduced to, explore their loss functions in greater detail, and work through some of the challenges that arise when training these models. 

Specifically, you will be able to experiment with different techniques for fine-tuning your models, explore the use of regularization to prevent overfitting, and consider the various trade-offs that arise when selecting different hyperparameters.

The exercises will help you to develop a deeper understanding of how to apply the techniques that you've learned in a practical setting, by providing you with hands-on experience in working with real-world data. The skills that you develop through this section will help you to become a more effective practitioner in the field of machine learning, and to take on more advanced challenges as you continue to progress in your studies.

Exercise 2.5.1: Implementing a Variational Autoencoder (VAE)

In this exercise, you will implement a basic VAE in TensorFlow. The aim is to create a VAE that can generate new samples from the MNIST dataset.

Example:

Here's an example of how to implement a simple VAE in TensorFlow:

import tensorflow as tf
from tensorflow.keras import layers

class VAE(tf.keras.Model):
    def __init__(self, latent_dim):
        super(VAE, self).__init__()
        self.latent_dim = latent_dim
        self.encoder = tf.keras.Sequential([
            layers.InputLayer(input_shape=(28, 28, 1)),
            layers.Conv2D(filters=32, kernel_size=3, strides=(2, 2), activation='relu'),
            layers.Conv2D(filters=64, kernel_size=3, strides=(2, 2), activation='relu'),
            layers.Flatten(),
            layers.Dense(latent_dim + latent_dim),
        ])
        self.decoder = tf.keras.Sequential([
            layers.InputLayer(input_shape=(latent_dim,)),
            layers.Dense(units=7*7*32, activation=tf.nn.relu),
            layers.Reshape(target_shape=(7, 7, 32)),
            layers.Conv2DTranspose(filters=64, kernel_size=3, strides=2, padding='same', activation='relu'),
            layers.Conv2DTranspose(filters=32, kernel_size=3, strides=2, padding='same', activation='relu'),
            layers.Conv2DTranspose(filters=1, kernel_size=3, strides=1, padding='same'),
        ])

    def encode(self, x):
        mean, logvar = tf.split(self.encoder(x), num_or_size_splits=2, axis=1)
        return mean, logvar

    def reparameterize(self, mean, logvar):
        epsilon = tf.random.normal(shape=mean.shape)
        return mean + tf.exp(0.5 * logvar) * epsilon

    def decode(self, z, apply_sigmoid=False):
        logits = self.decoder(z)
        if apply_sigmoid:
            probs = tf.sigmoid(logits)
            return probs
        return logits

    def call(self, inputs):
        mean, logvar = self.encode(inputs)
        z = self.reparameterize(mean, logvar)
        return self.decode(z)

Exercise 2.5.2: Implementing a Generative Adversarial Network (GAN)

Implement a GAN to generate new samples from the MNIST dataset. Pay particular attention to the loss functions of the generator and discriminator.

Example:

Here's an example of how to implement a simple GAN in TensorFlow:

import tensorflow as tf
from tensorflow.keras import layers

def make_generator_model():
    model = tf.keras.Sequential()
    model.add(layers.Dense(7*7*256, use_bias=False, input_shape=(100,)))
    model.add(layers.BatchNormalization())
    model.add(layers.LeakyReLU())

    model.add(layers.Reshape((7, 7, 256)))
    assert model.output_shape == (None, 7, 7, 256)  # Note: None is the batch size

    model.add(layers.Conv2DTranspose(128, (5, 5), strides=(1, 1), padding='same', use_bias=False))
    assert model.output_shape == (None, 7, 7, 128)
    model.add(layers.BatchNormalization())
    model.add(layers.LeakyReLU())

    model.add(layers.Conv2DTranspose(64, (5, 5), strides=(2, 2), padding='same', use_bias=False))
    assert model.output_shape == (None, 14, 14, 64)
    model.add(layers.BatchNormalization())
    model.add(layers.LeakyReLU())

    model.add(layers.Conv2DTranspose(1, (5, 5), strides=(2, 2), padding='same', use_bias=False, activation='tanh'))
    assert model.output_shape == (None, 28, 28, 1)

    return model

def make_discriminator_model():
    model = tf.keras.Sequential()
    model.add(layers.Conv2D(64, (5, 5), strides=(2, 2), padding='same', input_shape=[28, 28, 1]))
    model.add(layers.LeakyReLU())
    model.add(layers.Dropout(0.3))

    model.add(layers.Conv2D(128, (5, 5), strides=(2, 2), padding='same'))
    model.add(layers.LeakyReLU())
    model.add(layers.Dropout(0.3))

    model.add(layers.Flatten())
    model.add(layers.Dense(1))

    return model

Exercise 2.5.3: Experimenting with Loss Functions

In this exercise, you'll modify the GAN you implemented in Exercise 2.5.2 to use a different loss function. Specifically, implement the least squares GAN loss and observe any differences in training and output quality. 

Example:

Here's an example of how to implement the least squares GAN loss:

def generator_loss_LSGAN(fake_output):
    return tf.reduce_mean((fake_output - 1) ** 2)

def discriminator_loss_LSGAN(real_output, fake_output):
    return 0.5 * (tf.reduce_mean((real_output - 1) ** 2) + tf.reduce_mean(fake_output ** 2))

Exercise 2.5.4: Mode Collapse and Potential Solutions

In this exercise, we'll intentionally cause mode collapse in a GAN and then attempt to mitigate it. First, train a GAN on a diverse dataset but restrict the output size so that the GAN cannot reproduce the diversity of the input data. Observe the mode collapse. Then, implement one of the potential solutions to mode collapse discussed in this chapter, such as modifying the loss function or changing the model architecture. 

Remember to experiment and explore different parameters, architectures, and techniques. The more you experiment, the better you'll understand the nuances of training generative models.

In the next chapter, we will go deeper into specific types of generative models and their applications. Stay tuned! 

Chapter 2 Conclusion

In this chapter, we delved into the fascinating world of generative models. We started by understanding the concept of generative models and their importance in the world of deep learning. These models, as we learned, are capable of generating new, unseen data that share the same underlying patterns as the training data. This has far-reaching implications for a variety of fields, including art, entertainment, healthcare, and more.

We then looked at different types of generative models, including Variational Autoencoders (VAEs), Generative Adversarial Networks (GANs), and Autoregressive models. Each type has its unique strengths and characteristics that make them suitable for different types of problems.

Next, we dug into the nuances of training generative models, exploring concepts such as the likelihood, maximum likelihood estimation, and the evidence lower bound (ELBO). We learned how these concepts play a role in training our models and ensuring that they learn a robust representation of the data distribution.

Finally, we addressed some of the challenges that come with training generative models. Issues such as mode collapse and vanishing gradients can hamper the performance of our models, but we also discussed potential solutions to these problems.

The practical exercises at the end of this chapter were designed to help you apply these theoretical concepts. By implementing and experimenting with different types of generative models, you gain hands-on experience and a deeper understanding of the material.

In the next chapter, we'll dive into more details about Generative Adversarial Networks (GANs). We will explore their architecture, how they work, and how to implement them in TensorFlow. Stay tuned for more exciting insights into the world of generative models!

2.5 Practical Exercises of Chapter 2: Understanding Generative Models

The exercises in this section will provide you with ample opportunity to apply the concepts that you've learned in this chapter. You will have the chance to delve more deeply into the various types of generative models that you've been introduced to, explore their loss functions in greater detail, and work through some of the challenges that arise when training these models. 

Specifically, you will be able to experiment with different techniques for fine-tuning your models, explore the use of regularization to prevent overfitting, and consider the various trade-offs that arise when selecting different hyperparameters.

The exercises will help you to develop a deeper understanding of how to apply the techniques that you've learned in a practical setting, by providing you with hands-on experience in working with real-world data. The skills that you develop through this section will help you to become a more effective practitioner in the field of machine learning, and to take on more advanced challenges as you continue to progress in your studies.

Exercise 2.5.1: Implementing a Variational Autoencoder (VAE)

In this exercise, you will implement a basic VAE in TensorFlow. The aim is to create a VAE that can generate new samples from the MNIST dataset.

Example:

Here's an example of how to implement a simple VAE in TensorFlow:

import tensorflow as tf
from tensorflow.keras import layers

class VAE(tf.keras.Model):
    def __init__(self, latent_dim):
        super(VAE, self).__init__()
        self.latent_dim = latent_dim
        self.encoder = tf.keras.Sequential([
            layers.InputLayer(input_shape=(28, 28, 1)),
            layers.Conv2D(filters=32, kernel_size=3, strides=(2, 2), activation='relu'),
            layers.Conv2D(filters=64, kernel_size=3, strides=(2, 2), activation='relu'),
            layers.Flatten(),
            layers.Dense(latent_dim + latent_dim),
        ])
        self.decoder = tf.keras.Sequential([
            layers.InputLayer(input_shape=(latent_dim,)),
            layers.Dense(units=7*7*32, activation=tf.nn.relu),
            layers.Reshape(target_shape=(7, 7, 32)),
            layers.Conv2DTranspose(filters=64, kernel_size=3, strides=2, padding='same', activation='relu'),
            layers.Conv2DTranspose(filters=32, kernel_size=3, strides=2, padding='same', activation='relu'),
            layers.Conv2DTranspose(filters=1, kernel_size=3, strides=1, padding='same'),
        ])

    def encode(self, x):
        mean, logvar = tf.split(self.encoder(x), num_or_size_splits=2, axis=1)
        return mean, logvar

    def reparameterize(self, mean, logvar):
        epsilon = tf.random.normal(shape=mean.shape)
        return mean + tf.exp(0.5 * logvar) * epsilon

    def decode(self, z, apply_sigmoid=False):
        logits = self.decoder(z)
        if apply_sigmoid:
            probs = tf.sigmoid(logits)
            return probs
        return logits

    def call(self, inputs):
        mean, logvar = self.encode(inputs)
        z = self.reparameterize(mean, logvar)
        return self.decode(z)

Exercise 2.5.2: Implementing a Generative Adversarial Network (GAN)

Implement a GAN to generate new samples from the MNIST dataset. Pay particular attention to the loss functions of the generator and discriminator.

Example:

Here's an example of how to implement a simple GAN in TensorFlow:

import tensorflow as tf
from tensorflow.keras import layers

def make_generator_model():
    model = tf.keras.Sequential()
    model.add(layers.Dense(7*7*256, use_bias=False, input_shape=(100,)))
    model.add(layers.BatchNormalization())
    model.add(layers.LeakyReLU())

    model.add(layers.Reshape((7, 7, 256)))
    assert model.output_shape == (None, 7, 7, 256)  # Note: None is the batch size

    model.add(layers.Conv2DTranspose(128, (5, 5), strides=(1, 1), padding='same', use_bias=False))
    assert model.output_shape == (None, 7, 7, 128)
    model.add(layers.BatchNormalization())
    model.add(layers.LeakyReLU())

    model.add(layers.Conv2DTranspose(64, (5, 5), strides=(2, 2), padding='same', use_bias=False))
    assert model.output_shape == (None, 14, 14, 64)
    model.add(layers.BatchNormalization())
    model.add(layers.LeakyReLU())

    model.add(layers.Conv2DTranspose(1, (5, 5), strides=(2, 2), padding='same', use_bias=False, activation='tanh'))
    assert model.output_shape == (None, 28, 28, 1)

    return model

def make_discriminator_model():
    model = tf.keras.Sequential()
    model.add(layers.Conv2D(64, (5, 5), strides=(2, 2), padding='same', input_shape=[28, 28, 1]))
    model.add(layers.LeakyReLU())
    model.add(layers.Dropout(0.3))

    model.add(layers.Conv2D(128, (5, 5), strides=(2, 2), padding='same'))
    model.add(layers.LeakyReLU())
    model.add(layers.Dropout(0.3))

    model.add(layers.Flatten())
    model.add(layers.Dense(1))

    return model

Exercise 2.5.3: Experimenting with Loss Functions

In this exercise, you'll modify the GAN you implemented in Exercise 2.5.2 to use a different loss function. Specifically, implement the least squares GAN loss and observe any differences in training and output quality. 

Example:

Here's an example of how to implement the least squares GAN loss:

def generator_loss_LSGAN(fake_output):
    return tf.reduce_mean((fake_output - 1) ** 2)

def discriminator_loss_LSGAN(real_output, fake_output):
    return 0.5 * (tf.reduce_mean((real_output - 1) ** 2) + tf.reduce_mean(fake_output ** 2))

Exercise 2.5.4: Mode Collapse and Potential Solutions

In this exercise, we'll intentionally cause mode collapse in a GAN and then attempt to mitigate it. First, train a GAN on a diverse dataset but restrict the output size so that the GAN cannot reproduce the diversity of the input data. Observe the mode collapse. Then, implement one of the potential solutions to mode collapse discussed in this chapter, such as modifying the loss function or changing the model architecture. 

Remember to experiment and explore different parameters, architectures, and techniques. The more you experiment, the better you'll understand the nuances of training generative models.

In the next chapter, we will go deeper into specific types of generative models and their applications. Stay tuned! 

Chapter 2 Conclusion

In this chapter, we delved into the fascinating world of generative models. We started by understanding the concept of generative models and their importance in the world of deep learning. These models, as we learned, are capable of generating new, unseen data that share the same underlying patterns as the training data. This has far-reaching implications for a variety of fields, including art, entertainment, healthcare, and more.

We then looked at different types of generative models, including Variational Autoencoders (VAEs), Generative Adversarial Networks (GANs), and Autoregressive models. Each type has its unique strengths and characteristics that make them suitable for different types of problems.

Next, we dug into the nuances of training generative models, exploring concepts such as the likelihood, maximum likelihood estimation, and the evidence lower bound (ELBO). We learned how these concepts play a role in training our models and ensuring that they learn a robust representation of the data distribution.

Finally, we addressed some of the challenges that come with training generative models. Issues such as mode collapse and vanishing gradients can hamper the performance of our models, but we also discussed potential solutions to these problems.

The practical exercises at the end of this chapter were designed to help you apply these theoretical concepts. By implementing and experimenting with different types of generative models, you gain hands-on experience and a deeper understanding of the material.

In the next chapter, we'll dive into more details about Generative Adversarial Networks (GANs). We will explore their architecture, how they work, and how to implement them in TensorFlow. Stay tuned for more exciting insights into the world of generative models!