Skip to content

Conversation

@kholdrex
Copy link
Member

@kholdrex kholdrex commented Jul 2, 2025

This PR adds a comprehensive learning rate scheduling system to the rust-lstm library, bringing it to feature parity with PyTorch and TensorFlow for adaptive learning rate management.

Why This Matters

Learning rate scheduling is critical for training quality in modern deep learning:

  • Improves convergence speed and final model performance
  • Prevents overfitting and training instability
  • Enables state-of-the-art training techniques like OneCycleLR
  • Essential for production-ready deep learning workflows

What's New

Core Features

  • 8 Different Schedulers: StepLR, MultiStepLR, ExponentialLR, CosineAnnealingLR, CosineAnnealingWarmRestarts, OneCycleLR, ReduceLROnPlateau, LinearLR
  • ScheduledOptimizer: Seamless integration between any optimizer and scheduler
  • ScheduledLSTMTrainer: Automatic scheduler stepping during training
  • Enhanced Metrics: Learning rate tracking in training history

Convenience Functions

// Quick setup with popular combinations
let trainer = create_one_cycle_trainer(network, 0.1, 100);
let trainer = create_step_lr_trainer(network, 0.01, 10, 0.5);
let trainer = create_cosine_annealing_trainer(network, 0.01, 50, 1e-6);

Advanced Manual Configuration

let scheduled_optimizer = ScheduledOptimizer::new(
    Adam::new(0.001),
    OneCycleLR::new(0.1, 100),
    0.001
);

@kholdrex kholdrex self-assigned this Jul 2, 2025
@kholdrex kholdrex added the enhancement New feature or request label Jul 2, 2025
@kholdrex kholdrex merged commit 37c0b0f into main Jul 2, 2025
1 check passed
@kholdrex kholdrex deleted the feature/learning-rate-scheduling branch July 2, 2025 22:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants