Understanding benchmark results

[amogkam]

Hi thanks for sharing your benchmarks!

I have some questions on benchmark results-- why is does quantization not result in more memory savings & inference speedup?

For example, if we look at the benchmark results for CogVideoX on H100:

1. With no comilation, bf16 takes 112 seconds while fp8 with no compilation takes 113 seconds. Is there no benefit with torchao quantization unless compilation is also used?
2. With compilation enabled, bf16 takes 87 seconds and 33 GB of memory vs. fp8+compilation takes 75 seconds and 23 GB of memory. Why is the memory savings not 2x since you're going from 16 bit to 8 bit? Is it because only linear layers are quantized, but not attention operation? Also the time savings is only ~0.24 seconds per step (12 seconds saved over 50 steps). I would expect inference time to be a lot faster with quantization?

Thanks!

[sayakpaul]

I have some questions on benchmark results-- why is does quantization not result in more memory savings & inference speedup?

There are some specific examples where this is the case. Your statement made it sound like it's THE CASE for quantization schemes that were tried in the repository. This can happen for a number of reasons:

• Unavailability of kernels
• Computation and shape constraints required by the quantization methods being used (INT4, for example)

You're taking CogVideoX as an example to base your findings. Please make it explicitly clear in your description otherwise it all sounds quite general when it's not.

Is there no benefit with torchao quantization unless compilation is also used?

Yeah torchao is best used with compile.

With compilation enabled, bf16 takes 87 seconds and 33 GB of memory vs. fp8+compilation takes 75 seconds and 23 GB of memory. Why is the memory savings not 2x since you're going from 16 bit to 8 bit? Is it because only linear layers are quantized, but not attention operation?

Maximum memory when you load an FP8 model directly should be reduced by about 2x. But inference memory may not unless native FP8 kernels are leveraged. This is because inference usually also involved upcasting the FP8 params to the compute-dtype (bfloat16, say).

Is it because only linear layers are quantized, but not attention operation? Also the time savings is only ~0.24 seconds per step (12 seconds saved over 50 steps). I would expect inference time to be a lot faster with quantization?

Are you additionally compiling the VAE? If so, that may explain why because the VAE is known to trigger recompilations. Have tried benchmarking just the transformer of Cog with the exact settings?

[amogkam]

Thanks for the reply! Yes I'm specifically looking at CogVideoX for now.

Are you additionally compiling the VAE? If so, that may explain why because the VAE is known to trigger recompilations. Have tried benchmarking just the transformer of Cog with the exact settings?

I followed what you have here: https://github.com/sayakpaul/diffusers-torchao/blob/main/inference/benchmark_video.py, which doesn't compile the VAE.

[sayakpaul]

Ccing @a-r-r-o-w in case I missed anything.