Performance issue ...

[kif]

Hi,
I am looking for a fast and portable CRC implementation for Python ... but I think I found a performance bug:

import numpy,zlib,fastcrc
a = numpy.random.random((2048,2048))
%timeit zlib.crc32(a)
26.4 ms ± 29 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
%timeit fastcrc.crc32.aixm(a.tobytes())
83.9 ms ± 8.78 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
%timeit pyFAI.ext.fastcrc.crc32(a)
7.59 ms ± 2.07 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

The implementation from pyFAI relies on the SSE4 instruction, thus not portable.

[akhilles]

FYI, crc-rs has a new (unreleased) implementation that's ~6x faster in micro-benchmarks: https://github.com/mrhooray/crc-rs#throughput-gibs. However, a portable implementation just won't be able to match the performance of a SIMD implementation.

[eliphatfs]

I also found this. On a 8MB block on the machine, fastcrc is 10x slower than zlib crc32.
Also there are too many CRC32 but the one matching zlib crc32 (and C# System.IO.Hashing) should be iso_hdlc, which information I wish can be found in the documentation.

In [15]: %timeit fastcrc.crc32.iso_hdlc(data)
22.5 ms ± 1.53 μs per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [16]: %timeit zlib.crc32(data)
3.79 ms ± 6.91 μs per loop (mean ± std. dev. of 7 runs, 100 loops each)

I think zlib is using a portable C implementation.