detach losses when logging #1254
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jveitchmichaelis
changed the title
jveitchmichaelis
force-pushed
the
mem-improvements
branch
from
fba83e9 to
d345902
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1254 +/- ##
==========================================
+ Coverage 87.73% 87.75% +0.01%
==========================================
Files 20 20
Lines 2716 2720 +4
==========================================
+ Hits 2383 2387 +4
Misses 333 333
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
jveitchmichaelis
force-pushed
the
mem-improvements
branch
2 times, most recently
from
3cc27e5 to
893adca
Compare
jveitchmichaelis
changed the title
jveitchmichaelis
force-pushed
the
mem-improvements
branch
2 times, most recently
from
57a0954 to
034dc85
Compare
jveitchmichaelis
force-pushed
the
mem-improvements
branch
5 times, most recently
from
42ad6e8 to
3e84a0a
Compare
This was referenced Jan 1, 2026
jveitchmichaelis
force-pushed
the
mem-improvements
branch
from
3e84a0a to
4f09db1
Compare
3 tasks
jveitchmichaelis
force-pushed
the
mem-improvements
branch
from
4f09db1 to
04fb3d1
Compare
jveitchmichaelis
changed the title
jveitchmichaelis
requested review from
bw4sz and
henrykironde
and removed request for
bw4sz
bw4sz
approved these changes
Jan 2, 2026
Collaborator
bw4sz
left a comment
bw4sz
left a comment
There was a problem hiding this comment.
I agree that pytorch lightning should be doing this, but I don't see a downside for us to be sure.
bw4sz
pushed a commit
that referenced
this pull request
Jan 11, 2026
bw4sz
pushed a commit
that referenced
this pull request
Jan 16, 2026
bw4sz
pushed a commit
that referenced
this pull request
Jan 16, 2026
bw4sz
pushed a commit
that referenced
this pull request
Jan 16, 2026
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
detachlosses when logging. It's probable thatlightningdoes this for us under the hood, but good to be clear.@bw4sz want to try these fixes with your bird model branch?
I have also tried garbage collection, but I'm not sure it's necessary. Better to identify where we might be leaking.
Related Issue(s)
#1247
AI-Assisted Development
Some sanity checking with Claude Code, and an assisted review of where we might be missing resets etc.