[QUESTION]NVFP4 Post SFT Training & Model Accuracy

[deepak-vij]

Your question
Hi @Phlip79 , this is regarding the issue I created earlier (see below) related to nvfp4. One of the follow on activities I am trying to do is to gauge the accuracy of the SFT trained model using nvfp4. After completion of the SFT training run using nvfp4, I tried accessing the model using SgLang inference serving engine. The output of the inference request seems to be all gibberish. It all seems to work fine without any quantization. Is there anything I am missing that is messing up the model while doing post SFT training using nvfp4? Thanks.

#3470 (comment)

[Phlip79]

I would need a lot more information to help you out. But I would start investigating by starting with higher precision formats and working your way down.

[deepak-vij]

Hi @Phlip79 , as I mentioned that I wanted to gauge the accuracy of the nvfp4 quantization aware SFT generated model versus non-quantization aware SFT model. My baseline model Qwen3-VL-8B-Instruct is of type bf16 and I am using the following input dataset for SFT training. The accuracy test I have, that uses SGLang as the inference server, works fine for the non-quantization aware SFT generated model. However, when I run the same accuracy script against the nvfp4 quantization aware SFT generated model, I see gibberish coming out of the model. I am wondering if there are any issues with bf16->nvfp4->bf16 conversion process and the model weights stored as bf16 at the completion of SFT process. I am getting this issue even for a very simple prompt - "weather in France".

Is there a way to isolate the issue, meaning is there a way to ascertain that the model weights are fine (no datatype issues etc.) without using Sglang? Let me know if I can provide more information. Thanks.

https://huggingface.co/datasets/chenhegu/geo3k_imgurl/viewer/default/train?row=0

[deepak-vij]

In bypassed SGLang and generated directly using transformers, the serving stack is eliminated (see below). This seems to indicate that The NVFP4 SFT model weights are degraded for some reason. So, this does not seem to be SGLang issue, I guess.

This is NOT an SGLang issue anymore.
+++++++++++++++++++++++
python - <<'PY'
import torch
from transformers import AutoProcessor, AutoModelForImageTextToText

path="/data04/deepak/final_model_nvfp4" # <-- change to your nvfp4 export dir
proc = AutoProcessor.from_pretrained(path, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
path, trust_remote_code=True, torch_dtype=torch.bfloat16, device_map="cuda"
)

messages = [{"role":"user","content":[{"type":"text","text":"weather in france"}]}]
prompt = proc.apply_chat_template(messages, add_generation_prompt=True)
inputs = proc.tokenizer(prompt, return_tensors="pt").to("cuda")

out = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(proc.tokenizer.decode(out[0], skip_special_tokens=True))
PY