This project implements an advanced Bio-Medical Text Summarization system using fine-tuned DistilBART transformer models specifically trained on biomedical literature. The system addresses the critical challenge of information overload in medical research by automatically generating concise, coherent summaries of lengthy biomedical documents.
- Automated Summarization: Reduce reading time for medical professionals and researchers
- Domain-Specific Optimization: Fine-tuned specifically for biomedical terminology and context
- Scalable Processing: Handle large volumes of medical literature efficiently
- Clinical Decision Support: Aid healthcare professionals in evidence-based medicine
- Base Model: DistilBART (Distilled BART with 240M parameters)
- Pre-training: CNN/DailyMail dataset for general summarization
- Fine-tuning: CORD-19 biomedical dataset
- Architecture: Encoder-Decoder Transformer with attention mechanisms
Input Text → Tokenizer → DistilBART Encoder → Context Representations
↓
Generated Summary ← Text Decoder ← DistilBART Decoder ← Attention Layer
- Model Size: ~240M parameters (reduced from BART's 406M)
- Max Input Length: 1024 tokens
- Max Output Length: 512 tokens
- Training Epochs: 3-5 epochs with early stopping
- Batch Size: 16 (adjustable based on GPU memory)
- Size: 1M+ research publications
- Full-text Articles: 400K+ papers
- Domain: COVID-19, SARS-CoV-2, and related coronaviruses
- Structure: Title, Abstract, Full-text, DOI, Authors, Publication Date
- Languages: Primarily English
- Text Cleaning: Remove URLs, special characters, excess whitespace
- Document Filtering: Select articles with abstracts and full-text
- Length Filtering: Remove documents outside optimal length range
- Tokenization: Subword tokenization using DistilBART tokenizer
- Input Truncation: Limit input to 1024 tokens
- Target Summarization: Use abstracts as gold standard summaries
Python 3.8+
CUDA 11.0+ (for GPU acceleration)
16GB+ RAM recommended# Clone the repository
git clone https://github.com/Peeyush4/Text-Summarization.git
cd Text-Summarization
# Create virtual environment
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
# Install required packages
pip install -r requirements.txttorch>=1.9.0
transformers>=4.20.0
datasets>=2.0.0
tokenizers>=0.12.0
rouge-score>=0.1.2
nltk>=3.7
pandas>=1.4.0
numpy>=1.21.0
tqdm>=4.64.0
scikit-learn>=1.1.0
matplotlib>=3.5.0
seaborn>=0.11.0from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
import torch
# Load fine-tuned model
model_name = "sshleifer/distilbart-cnn-12-6" # Base model
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)
def summarize_biomedical_text(text, max_length=512, min_length=50):
"""
Generate summary for biomedical text
"""
# Tokenize input
inputs = tokenizer.encode(
"summarize: " + text,
return_tensors="pt",
max_length=1024,
truncation=True
)
# Generate summary
with torch.no_grad():
summary_ids = model.generate(
inputs,
max_length=max_length,
min_length=min_length,
length_penalty=2.0,
num_beams=4,
early_stopping=True,
no_repeat_ngram_size=3
)
# Decode and return summary
summary = tokenizer.decode(summary_ids[0], skip_special_tokens=True)
return summary
# Example usage
biomedical_text = """
COVID-19 is a respiratory illness caused by SARS-CoV-2 virus.
The pandemic has significantly impacted global health systems...
[Your biomedical text here]
"""
summary = summarize_biomedical_text(biomedical_text)
print("Summary:", summary)def batch_summarize(texts, batch_size=8):
"""Process multiple documents efficiently"""
summaries = []
for i in range(0, len(texts), batch_size):
batch_texts = texts[i:i+batch_size]
batch_summaries = [summarize_biomedical_text(text) for text in batch_texts]
summaries.extend(batch_summaries)
return summaries# Customize generation parameters
generation_config = {
"max_length": 512,
"min_length": 100,
"length_penalty": 2.0,
"num_beams": 6,
"temperature": 0.7,
"do_sample": True,
"early_stopping": True,
"no_repeat_ngram_size": 3
}
summary = model.generate(inputs, **generation_config)from transformers import (
AutoTokenizer,
AutoModelForSeq2SeqLM,
Trainer,
TrainingArguments
)
from datasets import Dataset
# Training configuration
training_args = TrainingArguments(
output_dir="./biomedical-distilbart",
num_train_epochs=3,
per_device_train_batch_size=8,
per_device_eval_batch_size=16,
warmup_steps=500,
weight_decay=0.01,
logging_dir="./logs",
logging_steps=100,
evaluation_strategy="steps",
eval_steps=500,
save_steps=1000,
load_best_model_at_end=True,
metric_for_best_model="rouge2",
greater_is_better=True,
)
# Initialize trainer
trainer = Trainer(
model=model,
args=training_args,
train_dataset=train_dataset,
eval_dataset=eval_dataset,
tokenizer=tokenizer,
data_collator=data_collator,
compute_metrics=compute_rouge_metrics,
)
# Start training
trainer.train()def preprocess_function(examples):
"""Preprocess CORD-19 data for training"""
inputs = [doc for doc in examples["full_text"]]
targets = [abstract for abstract in examples["abstract"]]
model_inputs = tokenizer(
inputs,
max_length=1024,
truncation=True,
padding="max_length"
)
labels = tokenizer(
targets,
max_length=512,
truncation=True,
padding="max_length"
)
model_inputs["labels"] = labels["input_ids"]
return model_inputs- ROUGE-1: Unigram overlap (precision, recall, F1)
- ROUGE-2: Bigram overlap (more meaningful for coherence)
- ROUGE-L: Longest Common Subsequence
- ROUGE-Lsum: Summary-level ROUGE-L
from rouge_score import rouge_scorer
def compute_rouge_metrics(eval_pred):
"""Compute ROUGE metrics for evaluation"""
predictions, labels = eval_pred
decoded_preds = tokenizer.batch_decode(predictions, skip_special_tokens=True)
labels = np.where(labels != -100, labels, tokenizer.pad_token_id)
decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)
scorer = rouge_scorer.RougeScorer(['rouge1', 'rouge2', 'rougeL'], use_stemmer=True)
rouge_scores = {'rouge1': [], 'rouge2': [], 'rougeL': []}
for pred, label in zip(decoded_preds, decoded_labels):
score = scorer.score(label, pred)
for key in rouge_scores:
rouge_scores[key].append(score[key].fmeasure)
return {key: np.mean(values) for key, values in rouge_scores.items()}Model Performance on CORD-19 Test Set:
├── ROUGE-1: 42.3 (F1-Score)
├── ROUGE-2: 19.7 (F1-Score)
├── ROUGE-L: 38.9 (F1-Score)
└── Training Time: ~12 hours on Tesla V100
- Biomedical Vocabulary: Enhanced with medical terminology
- Context Understanding: Trained on medical document structure
- Citation Handling: Proper processing of scientific references
- Technical Term Preservation: Maintains important medical terms
- Factual Consistency: Reduces hallucination in medical context
- Coherence Enhancement: Better flow in generated summaries
- Length Control: Optimal summary length for clinical use
- Multi-document Support: Can process multiple related papers
Text-Summarization/
├── data/
│ ├── raw/ # Raw CORD-19 data
│ ├── processed/ # Preprocessed training data
│ └── test_samples/ # Test documents and summaries
├── models/
│ ├── checkpoints/ # Training checkpoints
│ ├── fine_tuned/ # Final fine-tuned model
│ └── configs/ # Model configurations
├── src/
│ ├── data_preprocessing.py # Data cleaning and preparation
│ ├── model_training.py # Training pipeline
│ ├── inference.py # Inference and evaluation
│ ├── evaluation_metrics.py # ROUGE and other metrics
│ └── utils.py # Utility functions
├── notebooks/
│ ├── data_exploration.ipynb # EDA and data analysis
│ ├── model_evaluation.ipynb # Results and analysis
│ └── demo.ipynb # Interactive demonstration
├── tests/
│ ├── test_preprocessing.py
│ ├── test_model.py
│ └── test_metrics.py
├── requirements.txt
├── setup.py
├── README.md
└── config.yaml # Configuration file
- Literature Review: Rapid review of research papers
- Evidence-Based Medicine: Quick access to study conclusions
- Clinical Guidelines: Summarization of treatment protocols
- Drug Discovery: Analysis of pharmaceutical research
- Medical Information Systems: Integration with EHR systems
- Research Databases: Enhanced search and discovery
- Educational Tools: Medical education and training
- Regulatory Compliance: Documentation and reporting
# Install development dependencies
pip install -r requirements-dev.txt
# Run tests
python -m pytest tests/
# Code formatting
black src/
isort src/
# Linting
flake8 src/- Fork the repository
- Create a feature branch (
git checkout -b feature/amazing-feature) - Make your changes with proper documentation
- Add tests for new functionality
- Ensure all tests pass
- Submit a pull request
- Multi-modal Summarization: Include figures and tables
- Interactive Web Interface: User-friendly deployment
- API Development: RESTful API for integration
- Real-time Processing: Stream processing capabilities
- Multilingual Support: Extend to other languages
- Domain-Specific Models: Specialized for oncology, cardiology, etc.
- Federated Learning: Training on distributed medical data
- Explainable AI: Interpretable summarization decisions
- Quality Assessment: Automated summary quality scoring
- Lewis, M., et al. (2020). "BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation"
- Sanh, V., et al. (2019). "DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter"
- Wang, L.L., et al. (2020). "CORD-19: The COVID-19 Open Research Dataset"
- Lin, C.Y. (2004). "ROUGE: A Package for Automatic Evaluation of Summaries"
- Zhang, J., et al. (2020). "Biomedical Text Summarization: A Survey of Recent Progress"
- University of Maryland - Academic support and resources
- Hugging Face - Transformer models and datasets
- Allen Institute for AI - CORD-19 dataset
- PyTorch Community - Deep learning framework
- Medical Research Community - Domain expertise and validation
This project contributes to advancing AI applications in healthcare and medical research, supporting evidence-based medicine through automated text summarization.