📦 Module 1: Data — The Fuel for LLMs

[malibayram]

📦 Module 1: Data — The Fuel for LLMs

This module focuses on understanding text data, tokenization, and preparing datasets for LLM training.

Tasks to Complete:

• Lesson 1.1 — Understanding Text & The Role of Tokenization

  • Explain different text units: words, subwords, characters
  • Introduce Byte Pair Encoding (BPE)
  • Demonstrate tokenization concepts with examples

• Lesson 1.2 — Practical Tokenization with tiktoken

  • Install and setup tiktoken
  • Implement encoding/decoding tokens
  • Work with vocabulary size and special tokens
  • Create hands-on tokenization examples

• Lesson 1.3 — Exploring Pretraining Datasets (WikiText / OpenWebText)

  • Download and explore WikiText dataset
  • Analyze dataset structure and content
  • Compare different pretraining datasets
  • Data preprocessing techniques

• Lesson 1.4 — Preparing Inputs & Targets

  • Create (input, target) pairs for next-token prediction
  • Implement block_size and batch_size management
  • Handle sequence padding and truncation

• Lesson 1.5 — Efficient Data Handling with PyTorch DataLoader

  • Create custom Dataset class
  • Implement efficient batching logic
  • Setup DataLoader with proper configurations

Deliverables:

• 5 video lectures (~25 minutes each)
• Tokenization notebook with examples
• Dataset preparation notebook
• Custom Dataset and DataLoader implementation
• Module quiz

Resources:

• tiktoken documentation
• WikiText dataset
• PyTorch DataLoader tutorials

[tllmmaster]

Hi Mehmet Ali BAYRAM, huge fan of your 'LLM from scratch' series! I'm applying what I've learned to build an LLM for the Turkmen language, but I've hit a specific roadmap hurdle regarding tokenizer efficiency.

I am training a BPE tokenizer (32k vocab) on a 1GB dataset for an agglutinative language (Turkmen). I noticed significant redundancy in the vocabulary (e.g., separate tokens for 'Turkmenistan', 'türkmenistan', 'TURKMENISTAN'), which wastes valuable vocabulary slots.I was tempted to manually prune or merge these entries in the JSON file post-training, but I know this might break the BPE merge rules. What is the best practice here?

[malibayram]

Hi @tllmmaster you may check our better solution for tokenization: https://arxiv.org/abs/2508.14292
İmplementation: https://github.com/malibayram/turkish-tokenizer
Here is a demo: https://huggingface.co/spaces/alibayram/turkish_tiktokenizer

[tllmmaster]

Thanks!

[malibayram]

You are most welcome and please do not hesitate if you need more details.