1. How to Train GPT
Four steps:
- Pretraining
Competition work
- Supervised Finetuning
SFL models
- Reward Modeling
RM Models
- Reinforcement Learning
RL Models
Data Collection
from arxiv wikipedia
Tokenization
Transform all text into one very long list of integers
Here is the example:
Typical Algorithm: Byte Pair Encoding
Byte Pair Encoding (BPE)
Because of the rapidly increasing number of vocabulary, the vocabulary list of the model is bound to grow to a very frightening order of magnitude.
The main idea of BPE: Each step replaces the most common pair of adjacent data units with a new unit that has not appeared in the data before, iterating repeatedly until the stop condition is met.
Here is the Example:
We get a list composed of letters "aaabdabaaaac". The first step is that we find the most common pair of adjacent data unit "aa" and we replace it with letter "Z". So we will get a new list composed of letters "ZabdabZZc". Then , in the second step, we will repeat the first again and use the letter "Y" in place of the unit "ab". Therefore, we obtain a new list of letters "ZYdYZZc". And so we just need seven letters to represent the the primary list, which composed of twelve letters.
By using BPE we can tokenize the raw text.
Pretraining
Training sequences are laid out as rows, delimited by special <|endoftext|> tokens.
The shape of arrays are (B, T). The B is the batch size and T is the maximum context length.
The goal of pretraining is to train the model can predict the next word based on the word before this word.
Here is sth need to pay attention:
- Base model doesn't answer questions
- It only wants to complete internet documents
- But it can be tricked in to performing tasks with prompt engineering
Supervised Finetuning
Train the model to get the ability that get the ideal answer from the prompt.
SFT
Reward Modeling
Comparison the output from the model.
We can obtain the scores of different outputs by users. This process may look useless, but we can use these scores for the next step.
Reinforcement Learning
Train the model to predict tokens that are closer to the high-score outputs.
RLHF(Reinforcement Learning Human Feedback) has fewer variations, but base models have more entropy. So these two types of models suit different tasks.
Base models can be better in tasks where you have N examples of things and want to generate more things.
2. How to use the systems
GPT is like a token simulator
When we want to improve the effectiveness of GPT by using CoT, the model may produce bad answers even wrong answers. So there is an approach called self-consistency, which ensemble multiple attempts to aggregate final answers.
There are many variants of CoT, for example, the line CoT or Tree CoT.
LLM doesn't want to succeed and they just want to imitate!
By this picture, we can realize that we need to do some work on RAG(Retrieval-Augmented Generation).
Using json format.
