Menu iconMenu iconGenerative Deep Learning with Python
Generative Deep Learning with Python

Chapter 8: Project: Text Generation with Autoregressive Models

8.4 Generating New Text

Once our autoregressive model has been trained, we can use it to generate new text. Here's a basic approach: 

  1. Start with a seed sequence. This could be a single character, a word, or a phrase. It's used as the starting point for text generation.
  2. Predict the next character (or word). We give the seed sequence to the model and it predicts the next character (or word). We can use a technique called "temperature sampling" to add some randomness to the predictions. With a higher temperature, the model will make more diverse and surprising predictions. With a lower temperature, the predictions will be more conservative and likely to stick to common words and phrases.
  3. Update the seed sequence. We append the predicted character (or word) to the seed sequence. If the seed sequence is longer than the sequence length the model was trained on, we remove the first character (or word).
  4. Repeat the process. We keep predicting the next character (or word) and updating the seed sequence until we've generated the desired amount of text.

Here's how you could implement this in Python: 

def generate_text(model, seed_text, num_generate, temperature):
    input_sequence = [char_to_index[char] for char in seed_text]

    for _ in range(num_generate):
        input_sequence_padded = pad_sequences([input_sequence], maxlen=SEQUENCE_LENGTH, padding='pre')
        predictions = model.predict(input_sequence_padded)[0]
        next_index = sample_with_temperature(predictions, temperature)
        input_sequence.append(next_index)

    generated_text = ''.join([index_to_char[index] for index in input_sequence])
    return generated_text

In this function:

  • model is the trained autoregressive model.
  • seed_text is the initial seed sequence.
  • num_generate is the number of characters (or words) to generate.
  • temperature is a parameter controlling the randomness of predictions.

The function sample_with_temperature might look like this:

def sample_with_temperature(predictions, temperature):
    predictions = np.asarray(predictions).astype('float64')
    predictions = np.log(predictions) / temperature
    exp_predictions = np.exp(predictions)
    predictions = exp_predictions / np.sum(exp_predictions)
    probabilities = np.random.multinomial(1, predictions, 1)
    return np.argmax(probabilities)

This function applies the temperature to the predictions, normalizes them to form a probability distribution, then samples from this distribution to get the index of the next character (or word).

Remember, the generated text will be heavily dependent on your seed text and the 'temperature' value. Lower values will result in generated text that is more conservative and similar to the training data. Higher values will result in more diverse and creative outputs.

After you've generated the text, you can inspect it to see what the model has come up with!

8.4 Generating New Text

Once our autoregressive model has been trained, we can use it to generate new text. Here's a basic approach: 

  1. Start with a seed sequence. This could be a single character, a word, or a phrase. It's used as the starting point for text generation.
  2. Predict the next character (or word). We give the seed sequence to the model and it predicts the next character (or word). We can use a technique called "temperature sampling" to add some randomness to the predictions. With a higher temperature, the model will make more diverse and surprising predictions. With a lower temperature, the predictions will be more conservative and likely to stick to common words and phrases.
  3. Update the seed sequence. We append the predicted character (or word) to the seed sequence. If the seed sequence is longer than the sequence length the model was trained on, we remove the first character (or word).
  4. Repeat the process. We keep predicting the next character (or word) and updating the seed sequence until we've generated the desired amount of text.

Here's how you could implement this in Python: 

def generate_text(model, seed_text, num_generate, temperature):
    input_sequence = [char_to_index[char] for char in seed_text]

    for _ in range(num_generate):
        input_sequence_padded = pad_sequences([input_sequence], maxlen=SEQUENCE_LENGTH, padding='pre')
        predictions = model.predict(input_sequence_padded)[0]
        next_index = sample_with_temperature(predictions, temperature)
        input_sequence.append(next_index)

    generated_text = ''.join([index_to_char[index] for index in input_sequence])
    return generated_text

In this function:

  • model is the trained autoregressive model.
  • seed_text is the initial seed sequence.
  • num_generate is the number of characters (or words) to generate.
  • temperature is a parameter controlling the randomness of predictions.

The function sample_with_temperature might look like this:

def sample_with_temperature(predictions, temperature):
    predictions = np.asarray(predictions).astype('float64')
    predictions = np.log(predictions) / temperature
    exp_predictions = np.exp(predictions)
    predictions = exp_predictions / np.sum(exp_predictions)
    probabilities = np.random.multinomial(1, predictions, 1)
    return np.argmax(probabilities)

This function applies the temperature to the predictions, normalizes them to form a probability distribution, then samples from this distribution to get the index of the next character (or word).

Remember, the generated text will be heavily dependent on your seed text and the 'temperature' value. Lower values will result in generated text that is more conservative and similar to the training data. Higher values will result in more diverse and creative outputs.

After you've generated the text, you can inspect it to see what the model has come up with!

8.4 Generating New Text

Once our autoregressive model has been trained, we can use it to generate new text. Here's a basic approach: 

  1. Start with a seed sequence. This could be a single character, a word, or a phrase. It's used as the starting point for text generation.
  2. Predict the next character (or word). We give the seed sequence to the model and it predicts the next character (or word). We can use a technique called "temperature sampling" to add some randomness to the predictions. With a higher temperature, the model will make more diverse and surprising predictions. With a lower temperature, the predictions will be more conservative and likely to stick to common words and phrases.
  3. Update the seed sequence. We append the predicted character (or word) to the seed sequence. If the seed sequence is longer than the sequence length the model was trained on, we remove the first character (or word).
  4. Repeat the process. We keep predicting the next character (or word) and updating the seed sequence until we've generated the desired amount of text.

Here's how you could implement this in Python: 

def generate_text(model, seed_text, num_generate, temperature):
    input_sequence = [char_to_index[char] for char in seed_text]

    for _ in range(num_generate):
        input_sequence_padded = pad_sequences([input_sequence], maxlen=SEQUENCE_LENGTH, padding='pre')
        predictions = model.predict(input_sequence_padded)[0]
        next_index = sample_with_temperature(predictions, temperature)
        input_sequence.append(next_index)

    generated_text = ''.join([index_to_char[index] for index in input_sequence])
    return generated_text

In this function:

  • model is the trained autoregressive model.
  • seed_text is the initial seed sequence.
  • num_generate is the number of characters (or words) to generate.
  • temperature is a parameter controlling the randomness of predictions.

The function sample_with_temperature might look like this:

def sample_with_temperature(predictions, temperature):
    predictions = np.asarray(predictions).astype('float64')
    predictions = np.log(predictions) / temperature
    exp_predictions = np.exp(predictions)
    predictions = exp_predictions / np.sum(exp_predictions)
    probabilities = np.random.multinomial(1, predictions, 1)
    return np.argmax(probabilities)

This function applies the temperature to the predictions, normalizes them to form a probability distribution, then samples from this distribution to get the index of the next character (or word).

Remember, the generated text will be heavily dependent on your seed text and the 'temperature' value. Lower values will result in generated text that is more conservative and similar to the training data. Higher values will result in more diverse and creative outputs.

After you've generated the text, you can inspect it to see what the model has come up with!

8.4 Generating New Text

Once our autoregressive model has been trained, we can use it to generate new text. Here's a basic approach: 

  1. Start with a seed sequence. This could be a single character, a word, or a phrase. It's used as the starting point for text generation.
  2. Predict the next character (or word). We give the seed sequence to the model and it predicts the next character (or word). We can use a technique called "temperature sampling" to add some randomness to the predictions. With a higher temperature, the model will make more diverse and surprising predictions. With a lower temperature, the predictions will be more conservative and likely to stick to common words and phrases.
  3. Update the seed sequence. We append the predicted character (or word) to the seed sequence. If the seed sequence is longer than the sequence length the model was trained on, we remove the first character (or word).
  4. Repeat the process. We keep predicting the next character (or word) and updating the seed sequence until we've generated the desired amount of text.

Here's how you could implement this in Python: 

def generate_text(model, seed_text, num_generate, temperature):
    input_sequence = [char_to_index[char] for char in seed_text]

    for _ in range(num_generate):
        input_sequence_padded = pad_sequences([input_sequence], maxlen=SEQUENCE_LENGTH, padding='pre')
        predictions = model.predict(input_sequence_padded)[0]
        next_index = sample_with_temperature(predictions, temperature)
        input_sequence.append(next_index)

    generated_text = ''.join([index_to_char[index] for index in input_sequence])
    return generated_text

In this function:

  • model is the trained autoregressive model.
  • seed_text is the initial seed sequence.
  • num_generate is the number of characters (or words) to generate.
  • temperature is a parameter controlling the randomness of predictions.

The function sample_with_temperature might look like this:

def sample_with_temperature(predictions, temperature):
    predictions = np.asarray(predictions).astype('float64')
    predictions = np.log(predictions) / temperature
    exp_predictions = np.exp(predictions)
    predictions = exp_predictions / np.sum(exp_predictions)
    probabilities = np.random.multinomial(1, predictions, 1)
    return np.argmax(probabilities)

This function applies the temperature to the predictions, normalizes them to form a probability distribution, then samples from this distribution to get the index of the next character (or word).

Remember, the generated text will be heavily dependent on your seed text and the 'temperature' value. Lower values will result in generated text that is more conservative and similar to the training data. Higher values will result in more diverse and creative outputs.

After you've generated the text, you can inspect it to see what the model has come up with!