Decompress return UnexpectedEOF if Skippable frames written #129
Description
What version of Go are you using (go version)?
go version go1.19.3 linux/amd64
What did you do?
Zstd spec provide skippable-frames - frames that ignores by decompressor and allows to write any user metadata into archive (e.g. seekable table).
If I try to decompress archive with skippable frame, I get io.UnexpectedEOF. Here is an example:
import (
"bytes"
"encoding/binary"
"fmt"
"testing"
"github.com/DataDog/zstd"
)
func TestDataDog(t *testing.T) {
// write payload
payload := make([]byte, 1_000_001)
out, err := zstd.Compress(nil, payload)
if err != nil {
t.Fail()
}
// write skip frame
skipFrame := make([]byte, 12)
binary.LittleEndian.PutUint32(skipFrame, 0x184D2A50) // skip frame magic
binary.LittleEndian.PutUint32(skipFrame[4:], 4) // user data size
binary.LittleEndian.PutUint32(skipFrame[8:], 12345) // user data
out = append(out, skipFrame...)
decomp, err := zstd.Decompress(nil, out)
if err != nil {
fmt.Printf("error: %s\n", err) // got here "error: unexpected EOF"
t.Fail()
}
if !bytes.Equal(payload, decomp) {
t.Fail()
}
}Note, that payload size is not accidental, if it's small and can be processed by ZSTD_decompress (https://github.com/DataDog/zstd/blob/5f14d6af117fa84b37f99d4cde775e6039be6d3b/zstd.go#LL146C16-L146C16), zstd.Decompress will return no error. Problem occur then we fallback to read it with stream reader.
This is related to #112, but in this case I add a valid zstd frame