Towards Optimal Learning of Language Models

Obejctive

A: Learning Law. B: Empirical Evidence.

1 Environment

The following packages are required:

numpy
torch
matplotlib
wandb
cvxpy
transformers

Or simply:

pip3 install -r requirements.txt

2 Data

The training data of Perceptron Linear Classification and Transformer Language Modeling can be downloaded from this link. The data should be placed in data/linear and data/tinystory.

3 Learning Policy Optimization

cd src/
bash scripts/transformer/opt_policy.sh ${PATH_TO}/learning_law/src
bash scripts/perceptron/opt_policy.sh ${PATH_TO}/learning_law/src

4 Evaluating Optimized Learning Policy

cd src/
bash scripts/transformer/eval_policies.sh ${PATH_TO}/learning_law/src
bash scripts/perceptron/eval_policies.sh ${PATH_TO}/learning_law/src