gpt-from-scratch

Latest project progress repo

Components

Encoder Model

positional_encoding.py: Implements sinusoidal positional encoding and learnt positional encoding (meeting 1)

attention.py: Implements scaled dot product attention and multi head attention (meetings 2 & 3)

transformer.py: Implements decoder-only transformer based on GPT2

Bigram Model

bigram.py: implments a basic bigram model

train_bigram.py: trains the basic bigram