common/crypto/keccak: vendor in golang.org/x/crypto/sha3

[Sahil-4555]

The upstream golang.org/x/crypto/sha3 package has removed the assembly-optimized Keccak implementation. To avoid a performance regression in Erigon’s hashing hot paths, we maintain a local Keccak implementation that preserves the amd64 assembly code instead of relying on golang.org/x/crypto/sha3.

Refrence: ethereum/go-ethereum#33323
And: golang/go#75486

--

To verify that keccakf.go, keccakf_amd64.go, keccakf_amd64.s, and sha3.go are identical matches to golang.org/x/crypto@v0.43.0, the version right before the asm was removed.

$ gh api "repos/golang/crypto/contents/sha3/keccakf.go?ref=v0.43.0" --jq '.content' | base64 -d > /tmp/upstream_keccakf.go
$ gh api "repos/golang/crypto/contents/sha3/keccakf_amd64.go?ref=v0.43.0" --jq '.content' | base64 -d > /tmp/upstream_keccakf_amd64.go
$ gh api "repos/golang/crypto/contents/sha3/keccakf_amd64.s?ref=v0.43.0" --jq '.content' | base64 -d > /tmp/upstream_keccakf_amd64.s
$ gh api 'repos/golang/crypto/contents/sha3/sha3.go?ref=v0.43.0' --jq '.content' | base64 -d > /tmp/upstream_sha3.go

$ diff -u /tmp/upstream_keccakf_amd64.go crypto/keccak/keccakf_amd64.go
$ diff -u /tmp/upstream_keccakf_amd64.s crypto/keccak/keccakf_amd64.s
$ diff -u /tmp/upstream_legacy_keccakf.go crypto/keccak/keccakf.go
$ diff -u /tmp/upstream_sha3.go crypto/keccak/sha3.go

The hashes.go and sha3_test.go just consists of deletions of unrelated sha3 code.

$ gh api "repos/golang/crypto/contents/sha3/hashes.go?ref=v0.43.0" --jq '.content' | base64 -d > /tmp/upstream_hashes.go
$ gh api "repos/golang/crypto/contents/sha3/sha3_test.go?ref=v0.43.0" --jq '.content' | base64 -d > /tmp/upstream_sha3_test.go

$ diff -u /tmp/upstream_hashes.go crypto/keccak/hashes.go
$ diff -u /tmp/upstream_sha3_test.go crypto/keccak/sha3_test.go

You can check that the asm optimization is still working by running:

go test ./crypto/keccak/... -bench=. -benchmem -count=5 | tee bench_asm.txt
go test -tags=purego ./crypto/keccak/... -bench=. -benchmem -count=5 | tee bench_purego.txt
benchstat bench_purego.txt bench_asm.txt

And all internal uses were successfully ported to our vendored packaged.

$ grep -r 'golang.org/x/crypto/sha3' --include='*.go' .

[Sahil-4555]

@AskAlexSharov
We are still importing golang.org/x/crypto/sha3 in common/crypto/crypto_test.go for TestKeccak256HasherNew. This usage is limited to tests only and is used for output verification, not for production hashing paths. All runtime Keccak usage has been migrated to the local crypto/keccak package. need your feedback before doing any changes

[AskAlexSharov]

@Sahil-4555 thank you for picking it up. Will check