Small LLM Project

This project demonstrates how to build, train, and interact with a small Large Language Model (LLM) using a custom dataset and tokenizer.

Overview

• Custom Training: Train a language model on your own text data with a custom tokenizer.
• Model Checkpoints: Save and reuse model checkpoints for evaluation or further training.
• Web Demo: Interact with the trained model through a simple web interface.
• Evaluation: Test and explore the model’s outputs using scripts or notebooks.

How It Works

1. Prepare Data: Process your text data and set up a tokenizer.
2. Train Model: Run the training script to build your LLM.
3. Demo & Inference: Use the web app or scripts to interact with the model and see results.

Features

• End-to-end pipeline: data prep, training, inference, and demo.
• Customizable for different datasets and tasks.
• Simple web interface for easy testing.

---