Skip to content

Comments

Fix batch training padding mismatch handling#22

Merged
H-Chris233 merged 1 commit intomainfrom
codex/fix-training-logic-errors-and-improve-documentation-blh261
Jan 15, 2026
Merged

Fix batch training padding mismatch handling#22
H-Chris233 merged 1 commit intomainfrom
codex/fix-training-logic-errors-and-improve-documentation-blh261

Conversation

@H-Chris233
Copy link
Owner

Motivation

  • Batch training used padded Batch inputs while create_training_batches produced targets based on real token lengths, causing logits/target length mismatches and incorrect gradient application in train_monitored_batch.
  • Improve clarity for this educational project by documenting the per-sample real-length slicing approach so learners understand why PAD tokens must not participate in training.

Description

  • In src/llm.rs train_monitored_batch compute per-sample real_len from input_batch.attention_mask and slice input tokens to tokens[0..real_len] so forward/backward operate only on real tokens, collecting per-sample batch_targets accordingly.
  • Add a safety check to skip malformed samples when log_probs.nrows() != target_ids.len() and avoid crashing or applying mismatched gradients.
  • Remove the previous approach of zeroing gradients by PAD positions because samples are now sliced to exclude PAD before gradient computation.
  • Update docs/批量训练与动态掩码.md to explain the real-length slicing rule and why PAD must not participate in gradient computation.

Testing

  • No automated tests were executed as part of this change.

Codex Task

Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3afa7178c7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@H-Chris233 H-Chris233 merged commit f40ceae into main Jan 15, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant