Read this in other languages: English, 中文.
Continuously optimizing and updating...
🗓️ Changelog
- 📝 Added reward model training, PPO training, GRPO training
Click for more details
- 📝 Pretokenized corpus and updated cold start code.
- 📝 Uploaded tokenizer training weights (1.56 chars/token), adjusted tokenizer format and training method to align with the style of the Qwen2 tokenizer.
- 📝 Added LLM documentation covering full lifecycle technical points of LLMs.
- 📝 Added support for MoE models. Training resource usage is unstable. Tested model with Experts=8, experts_per_tok=4. GPU memory fluctuated from 60% to 94%, and single GPU utilization fluctuated from 0% to 100%. After reducing the batch_size, memory usage stabilized at around 90%, single GPU utilization stabilized above 90%, occasionally dropping to around 30%.
- 📝 Added support for training with jsonl files.
- 📝 Added support for training code using DeepSpeed. Testing showed a maximum batch_size increase of 38% and a training speed increase of 9.6%.
- 📝 Added pre-training code for dolly_llm. Conducted a pre-training test: model 0.6B, corpus 500M, using 46GB * 4 GPUs.
- ✅ Released dolly_llm as a pip package for installation.
- 📝 Standardized modeling and configuration using the transformers format, and designed/modified modeling_dolly v0.1 with 11.5B parameters.
- 📝 Implemented the configuration_dolly class and added modeling_dolly v0.0.
- 📝 Added BBPE method training for the Tokenizer.
- 📝 Added tokenizer construction code, supporting sentencepiece and transformers' BPE. Supports building from text and from existing tokenizers.
- ✅ Tested constructing custom LLM model architectures from transformers.
Special thanks to the following resources and articles for their assistance: