model specifications & related logics

[shivendrra]

to be added:

• BERT language model
• decoder-only model with RMSNorm, RoPE, SwiGLU
• config file for different model size configs
• run file
• quantization logic
• fine-tuning script for model

to be done:

• test run once with smaller data
• hyperparameter tuning