Chapter 8: Project: Text Generation with Autoregressive Models
8.4 Generating New Text
Once our autoregressive model has been trained, we can use it to generate new text. Here's a basic approach:
- Start with a seed sequence. This could be a single character, a word, or a phrase. It's used as the starting point for text generation.
- Predict the next character (or word). We give the seed sequence to the model and it predicts the next character (or word). We can use a technique called "temperature sampling" to add some randomness to the predictions. With a higher temperature, the model will make more diverse and surprising predictions. With a lower temperature, the predictions will be more conservative and likely to stick to common words and phrases.
- Update the seed sequence. We append the predicted character (or word) to the seed sequence. If the seed sequence is longer than the sequence length the model was trained on, we remove the first character (or word).
- Repeat the process. We keep predicting the next character (or word) and updating the seed sequence until we've generated the desired amount of text.
Here's how you could implement this in Python:
def generate_text(model, seed_text, num_generate, temperature):
input_sequence = [char_to_index[char] for char in seed_text]
for _ in range(num_generate):
input_sequence_padded = pad_sequences([input_sequence], maxlen=SEQUENCE_LENGTH, padding='pre')
predictions = model.predict(input_sequence_padded)[0]
next_index = sample_with_temperature(predictions, temperature)
input_sequence.append(next_index)
generated_text = ''.join([index_to_char[index] for index in input_sequence])
return generated_textIn this function:
modelis the trained autoregressive model.seed_textis the initial seed sequence.num_generateis the number of characters (or words) to generate.temperatureis a parameter controlling the randomness of predictions.
The function sample_with_temperature might look like this:
def sample_with_temperature(predictions, temperature):
predictions = np.asarray(predictions).astype('float64')
predictions = np.log(predictions) / temperature
exp_predictions = np.exp(predictions)
predictions = exp_predictions / np.sum(exp_predictions)
probabilities = np.random.multinomial(1, predictions, 1)
return np.argmax(probabilities)This function applies the temperature to the predictions, normalizes them to form a probability distribution, then samples from this distribution to get the index of the next character (or word).
Remember, the generated text will be heavily dependent on your seed text and the 'temperature' value. Lower values will result in generated text that is more conservative and similar to the training data. Higher values will result in more diverse and creative outputs.
After you've generated the text, you can inspect it to see what the model has come up with!
8.4 Generating New Text
Once our autoregressive model has been trained, we can use it to generate new text. Here's a basic approach:
- Start with a seed sequence. This could be a single character, a word, or a phrase. It's used as the starting point for text generation.
- Predict the next character (or word). We give the seed sequence to the model and it predicts the next character (or word). We can use a technique called "temperature sampling" to add some randomness to the predictions. With a higher temperature, the model will make more diverse and surprising predictions. With a lower temperature, the predictions will be more conservative and likely to stick to common words and phrases.
- Update the seed sequence. We append the predicted character (or word) to the seed sequence. If the seed sequence is longer than the sequence length the model was trained on, we remove the first character (or word).
- Repeat the process. We keep predicting the next character (or word) and updating the seed sequence until we've generated the desired amount of text.
Here's how you could implement this in Python:
def generate_text(model, seed_text, num_generate, temperature):
input_sequence = [char_to_index[char] for char in seed_text]
for _ in range(num_generate):
input_sequence_padded = pad_sequences([input_sequence], maxlen=SEQUENCE_LENGTH, padding='pre')
predictions = model.predict(input_sequence_padded)[0]
next_index = sample_with_temperature(predictions, temperature)
input_sequence.append(next_index)
generated_text = ''.join([index_to_char[index] for index in input_sequence])
return generated_textIn this function:
modelis the trained autoregressive model.seed_textis the initial seed sequence.num_generateis the number of characters (or words) to generate.temperatureis a parameter controlling the randomness of predictions.
The function sample_with_temperature might look like this:
def sample_with_temperature(predictions, temperature):
predictions = np.asarray(predictions).astype('float64')
predictions = np.log(predictions) / temperature
exp_predictions = np.exp(predictions)
predictions = exp_predictions / np.sum(exp_predictions)
probabilities = np.random.multinomial(1, predictions, 1)
return np.argmax(probabilities)This function applies the temperature to the predictions, normalizes them to form a probability distribution, then samples from this distribution to get the index of the next character (or word).
Remember, the generated text will be heavily dependent on your seed text and the 'temperature' value. Lower values will result in generated text that is more conservative and similar to the training data. Higher values will result in more diverse and creative outputs.
After you've generated the text, you can inspect it to see what the model has come up with!
8.4 Generating New Text
Once our autoregressive model has been trained, we can use it to generate new text. Here's a basic approach:
- Start with a seed sequence. This could be a single character, a word, or a phrase. It's used as the starting point for text generation.
- Predict the next character (or word). We give the seed sequence to the model and it predicts the next character (or word). We can use a technique called "temperature sampling" to add some randomness to the predictions. With a higher temperature, the model will make more diverse and surprising predictions. With a lower temperature, the predictions will be more conservative and likely to stick to common words and phrases.
- Update the seed sequence. We append the predicted character (or word) to the seed sequence. If the seed sequence is longer than the sequence length the model was trained on, we remove the first character (or word).
- Repeat the process. We keep predicting the next character (or word) and updating the seed sequence until we've generated the desired amount of text.
Here's how you could implement this in Python:
def generate_text(model, seed_text, num_generate, temperature):
input_sequence = [char_to_index[char] for char in seed_text]
for _ in range(num_generate):
input_sequence_padded = pad_sequences([input_sequence], maxlen=SEQUENCE_LENGTH, padding='pre')
predictions = model.predict(input_sequence_padded)[0]
next_index = sample_with_temperature(predictions, temperature)
input_sequence.append(next_index)
generated_text = ''.join([index_to_char[index] for index in input_sequence])
return generated_textIn this function:
modelis the trained autoregressive model.seed_textis the initial seed sequence.num_generateis the number of characters (or words) to generate.temperatureis a parameter controlling the randomness of predictions.
The function sample_with_temperature might look like this:
def sample_with_temperature(predictions, temperature):
predictions = np.asarray(predictions).astype('float64')
predictions = np.log(predictions) / temperature
exp_predictions = np.exp(predictions)
predictions = exp_predictions / np.sum(exp_predictions)
probabilities = np.random.multinomial(1, predictions, 1)
return np.argmax(probabilities)This function applies the temperature to the predictions, normalizes them to form a probability distribution, then samples from this distribution to get the index of the next character (or word).
Remember, the generated text will be heavily dependent on your seed text and the 'temperature' value. Lower values will result in generated text that is more conservative and similar to the training data. Higher values will result in more diverse and creative outputs.
After you've generated the text, you can inspect it to see what the model has come up with!
8.4 Generating New Text
Once our autoregressive model has been trained, we can use it to generate new text. Here's a basic approach:
- Start with a seed sequence. This could be a single character, a word, or a phrase. It's used as the starting point for text generation.
- Predict the next character (or word). We give the seed sequence to the model and it predicts the next character (or word). We can use a technique called "temperature sampling" to add some randomness to the predictions. With a higher temperature, the model will make more diverse and surprising predictions. With a lower temperature, the predictions will be more conservative and likely to stick to common words and phrases.
- Update the seed sequence. We append the predicted character (or word) to the seed sequence. If the seed sequence is longer than the sequence length the model was trained on, we remove the first character (or word).
- Repeat the process. We keep predicting the next character (or word) and updating the seed sequence until we've generated the desired amount of text.
Here's how you could implement this in Python:
def generate_text(model, seed_text, num_generate, temperature):
input_sequence = [char_to_index[char] for char in seed_text]
for _ in range(num_generate):
input_sequence_padded = pad_sequences([input_sequence], maxlen=SEQUENCE_LENGTH, padding='pre')
predictions = model.predict(input_sequence_padded)[0]
next_index = sample_with_temperature(predictions, temperature)
input_sequence.append(next_index)
generated_text = ''.join([index_to_char[index] for index in input_sequence])
return generated_textIn this function:
modelis the trained autoregressive model.seed_textis the initial seed sequence.num_generateis the number of characters (or words) to generate.temperatureis a parameter controlling the randomness of predictions.
The function sample_with_temperature might look like this:
def sample_with_temperature(predictions, temperature):
predictions = np.asarray(predictions).astype('float64')
predictions = np.log(predictions) / temperature
exp_predictions = np.exp(predictions)
predictions = exp_predictions / np.sum(exp_predictions)
probabilities = np.random.multinomial(1, predictions, 1)
return np.argmax(probabilities)This function applies the temperature to the predictions, normalizes them to form a probability distribution, then samples from this distribution to get the index of the next character (or word).
Remember, the generated text will be heavily dependent on your seed text and the 'temperature' value. Lower values will result in generated text that is more conservative and similar to the training data. Higher values will result in more diverse and creative outputs.
After you've generated the text, you can inspect it to see what the model has come up with!