Training stage 1 error

[IceIce1ce]

Hi author, thanks for your great code. When I run your training code, I found that two annotations from mdetr do not include segmentation masks (final_flickr_separateGT_train.json and final_flickr_separateGT_val.json). This cause the training has an error. Can you check your training code again?

[IceIce1ce]

In addition, the code has another problem:

Traceback (most recent call last):
File "/home/vsw/Desktop/LLaVA-Grounding/llava/train/train_joint_1st.py", line 992, in
train()
File "/home/vsw/Desktop/LLaVA-Grounding/llava/train/train_joint_1st.py", line 970, in train
trainer.train()
File "/home/vsw/Desktop/LLaVA-Grounding/llava/train/llava_trainer_joint_train.py", line 1464, in train
return inner_training_loop(
File "/home/vsw/Desktop/LLaVA-Grounding/llava/train/llava_trainer_joint_train.py", line 1730, in _inner_training_loop
tr_loss_step = self.training_step(model, inputs)
File "/home/vsw/Desktop/LLaVA-Grounding/llava/train/llava_trainer_joint_train.py", line 2592, in training_step
loss = self.compute_loss(model, inputs)
File "/home/vsw/Desktop/LLaVA-Grounding/llava/train/llava_trainer_joint_train.py", line 2620, in compute_loss
outputs = model(**inputs)
File "/home/vsw/miniconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/home/vsw/miniconda3/envs/llava/lib/python3.10/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
ret_val = func(*args, **kwargs)
File "/home/vsw/miniconda3/envs/llava/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 1735, in forward
loss = self.module(*inputs, **kwargs)
File "/home/vsw/miniconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/home/vsw/Desktop/LLaVA-Grounding/llava/model/language_model/llava_llama_gd.py", line 489, in forward
return self.forward_inner(**llava_inputs)
File "/home/vsw/Desktop/LLaVA-Grounding/llava/model/language_model/llava_llama_gd.py", line 564, in forward_inner
ground_loss=self.seg_model(seg_inputs)
File "/home/vsw/miniconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/home/vsw/Desktop/LLaVA-Grounding/llava/model/openseed/BaseModel.py", line 53, in forward
outputs = self.model(*inputs, **kwargs)
File "/home/vsw/miniconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/home/vsw/Desktop/LLaVA-Grounding/llava/model/openseed/architectures/openseed_model.py", line 282, in forward
losses_flickr = self.forward_seg(batched_inputs['flickr'], task='seg',default_text_embeddings=batched_inputs['flickr_text_embeddings'],data_type='gd')
File "/home/vsw/Desktop/LLaVA-Grounding/llava/model/openseed/architectures/openseed_model.py", line 360, in forward_seg
targets = self.prepare_targets(gt_instances, images, task=task)
File "/home/vsw/Desktop/LLaVA-Grounding/llava/model/openseed/architectures/openseed_model.py", line 550, in prepare_targets
gt_masks = targets_per_image.gt_masks
File "/home/vsw/miniconda3/envs/llava/lib/python3.10/site-packages/detectron2/structures/instances.py", line 65, in getattr
raise AttributeError("Cannot find field '{}' in the given Instances!".format(name))
AttributeError: Cannot find field 'gt_masks' in the given Instances!

[IceIce1ce]

Moreover, this file does not exist for training stage 3: instances_train2017_refcoco.json.