NVFP4 or FP8 support instead of INT?

[mitchins]

Given Blackwell supports NVPF4 and Ada supports FP8, wouldn't that be more stable than a scaled INT format?

[brontoguana]

Thanks for raising this, it's a good question and worth discussing.

Research shows that INT4 with calibrated quantization performs on par with FP4 formats. NVIDIA's own NVFP4 evaluation reports similar quality on average between the two, with the primary advantage being native hardware acceleration on Blackwell rather than precision. Our INT4 implementation uses AWQ with per-group scales (group size 128), which is well-proven across the ecosystem.

The constraints vary from model to model and per the individual hardware config, sometimes PCIE is the main constraint (for larger models especially) and sometimes compute is more significant.

NVFP4 could bring performance gains on some models for Blackwell chips so that's something I will keep open and consider for the future.