Skip to content

Labelmodel takes up a huge amount of memory when training, and it also wouldn't release the memory #19

@topwasu

Description

@topwasu

During the training, labelmodel takes up a lot of memory causing an out-of-memory- error, and reducing the batch size doesn't help. Apart from that, labelmodel seems to still take up memory even at the start of a new checkpoint. For now, in the master branch, the labelmodel is replaced with majority vote. In memory-debug branch, it is still using labelmodel, and it contains different memory profilers to help debug this bug, so debugging this error should probably happen in this branch.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions