Get much higher accuracy than those in your paper.

[clevercool]

Hi,

Thanks for your open-source of the AdaptivFloat. It is an impressive paper.

I implemented your code in Resnet-50 model with imageNet dataset.(https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py)
I quantized 54 layers (including downsample layers) only on weight:
enabling quant: conv1
enabling quant: layer1.0.conv1
enabling quant: layer1.0.conv2
enabling quant: layer1.0.conv3
enabling quant: layer1.0.downsample.0
enabling quant: layer1.1.conv1
enabling quant: layer1.1.conv2
enabling quant: layer1.1.conv3
enabling quant: layer1.2.conv1
enabling quant: layer1.2.conv2
enabling quant: layer1.2.conv3
enabling quant: layer2.0.conv1
enabling quant: layer2.0.conv2
enabling quant: layer2.0.conv3
enabling quant: layer2.0.downsample.0
enabling quant: layer2.1.conv1
enabling quant: layer2.1.conv2
enabling quant: layer2.1.conv3
enabling quant: layer2.2.conv1
enabling quant: layer2.2.conv2
enabling quant: layer2.2.conv3
enabling quant: layer2.3.conv1
enabling quant: layer2.3.conv2
enabling quant: layer2.3.conv3
enabling quant: layer3.0.conv1
enabling quant: layer3.0.conv2
enabling quant: layer3.0.conv3
enabling quant: layer3.0.downsample.0
enabling quant: layer3.1.conv1
enabling quant: layer3.1.conv2
enabling quant: layer3.1.conv3
enabling quant: layer3.2.conv1
enabling quant: layer3.2.conv2
enabling quant: layer3.2.conv3
enabling quant: layer3.3.conv1
enabling quant: layer3.3.conv2
enabling quant: layer3.3.conv3
enabling quant: layer3.4.conv1
enabling quant: layer3.4.conv2
enabling quant: layer3.4.conv3
enabling quant: layer3.5.conv1
enabling quant: layer3.5.conv2
enabling quant: layer3.5.conv3
enabling quant: layer4.0.conv1
enabling quant: layer4.0.conv2
enabling quant: layer4.0.conv3
enabling quant: layer4.0.downsample.0
enabling quant: layer4.1.conv1
enabling quant: layer4.1.conv2
enabling quant: layer4.1.conv3
enabling quant: layer4.2.conv1
enabling quant: layer4.2.conv2
enabling quant: layer4.2.conv3
enabling quant: fc

I only replace the weight tensors using the NumPy function in this repository.
However, the post-training results are much higher than those in your paper.
For example, the 4-bit and 5-bit results of mine are 49.25% and 69.56% (The source accuracy is 76.13% in PyTorch), the paper reported 29.0% and 67.2%.

Is there something about calibration I missed?
Could you provide more details about the implementation?

Thanks!

[ttambe]

Hi, what do you have in the n_exp argument for the 4-bit and 5-bit results?

[clevercool]

Hi, what do you have in the n_exp argument for the 4-bit and 5-bit results?

3 bits for all.