Skip to content

Conversation

@sebastian-nagel
Copy link
Contributor

  • upgrade to OkHttp 5.3.2
  • enable support for zstd content-encoding

Meta sites already support zstd content-encoding:

$> $NUTCH_HOME/bin/nutch parsechecker https://www.llama.com/faq/
...
  _request_ = GET /faq/ HTTP/2
...
Accept-Encoding: zstd, br, gzip
...
  _response.headers_ = HTTP/2 200 
vary: Accept-Encoding
content-encoding: zstd
...

(with http.useHttp2, store.http.request and store.http.headers set to true)

- upgrade to OkHttp 5.3.2
- enable support for zstd content-encoding
Copy link
Member

@lewismc lewismc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice. Also good to get a major dependency upgrade for the okhttp dependency.

@sebastian-nagel sebastian-nagel merged commit 00bf8c4 into apache:master Dec 18, 2025
6 checks passed
@sebastian-nagel sebastian-nagel deleted the NUTCH-3139 branch December 18, 2025 09:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants