The cleanest collection of modern LLM recipes (RoPE, GQA, RMSNorm, MoE, SSM, etc.) in nanoGPT style.
(👷♂️ under construction)
NanoGPX overwrites Andrej's nanoGPT, implementing state-of-the-art recipes inspired from modern LLM architectures such as Qwen3, LFM, etc. The code inherits nanoGPT's single-file and hackable style and are tested on the same training datasets (OpenWebText) for performance comparison.
- RoPE (Rotary Positional Embeddings)
- GQA (Grouped Query Attention)
- SwiGLU activation
- RMSNorm
- MoE (Mixture of Expert)
- Hybrid SSM (State Space Model)
- Muon Optimizer