Transformers-Implementation

An educational from-scratch implementation of the Transformer architecture in PyTorch, based on the paper "Attention Is All You Need".

This repository demonstrates how the encoder–decoder Transformer works by building all of its core components step by step.

---

🧰 Project Structure

Transformers-Implementation/
├── src/
│ ├── add_and_norm.py # Add & Norm layer
│ ├── attention.py # Scaled Dot-Product Attention + Multi-Head Attention
│ ├── decoder.py # Transformer Decoder
│ ├── encoder.py # Transformer Encoder
│ ├── positional_encoding.py # Sinusoidal Positional Encoding
│ ├── position_wise_fnn.py # Position-wise Feed-Forward Network
│ └── transformer.py # Full Transformer model
├── test/ # Unit tests
├── pytest.ini # Pytest configuration
├── .gitignore
└── README.md

---

🚀 Features

• Add & Norm Layer – residual connection + layer normalization.
• Scaled Dot-Product Attention with optional causal masking.
• Multi-Head Attention – parallel attention across multiple heads.
• Position-wise Feed-Forward Network – two linear layers with ReLU.
• Encoder & Decoder Layers – stacked with attention and feed-forward sublayers.
• Positional Encoding – sinusoidal encodings to inject sequence order.
• Full Transformer Model – combines encoder, decoder, and positional encoding.

---

🔧 Requirements

• Python 3.7+
• PyTorch
• pytest (for running tests)

Install dependencies:

pip install -r requirements.txt