Skip to content

Latest commit

 

History

History
34 lines (27 loc) · 1.36 KB

README.md

File metadata and controls

34 lines (27 loc) · 1.36 KB
Raw

A PyTorch implementation of generative pre-trained transformers (GPT)

This is a personal exercise to build, train and use GPT, inspired by minGPT.

Prerequisite

Installation

git clone https://github.com/rygx/simple-gpt.git && cd simple-gpt
python -m venv .venv
pip install pip-tools
./update_deps.sh

Usage

Data

Any text-fromat data file should be usable, given the size can fit with training and inference environment.

Train

Use train/train_gpt.py. After training, model state dictionary and hyper parameters will be stored in models directory.

Generate

Use train/generate.py. Under models/ directory there is already a coarsely pre-trained model (using minGPT's tinyshakespeare sample) with ID 9cdb42ed-0b16-4a3a-88e2-fffa61fa4f50. Generate texts using this model with the following command:

python train/generate.py --dir "models" -u "9cdb42ed-0b16-4a3a-88e2-fffa61fa4f50" --prompt "QUEEN: "

Generation length (in number of tokens) and temperature can also be tuned using --length/-l and --temp/-t options.

TODOs/plans

  • Easier setup (e.g., using setuptool)
  • Unit tests :P
  • Implement own tokenizer