[improve][broker] Update netty allocator default chunk size from 4MB to 8MB

[geniusjoe]

Main Issue: #25021

Motivation

Netty changed the default chunk size from 16M to 4M in version 4.1.76. The primary reason was that the Netty community believed that in most scenarios, each PoolArena does not require very large chunk sizes, as it could easily lead to wasted memory resources in this pr netty/netty#12108, , which I think is not very suitable for pulsar use case:

If we don't need a lot of memory, and certainly not compared to the number of cores on a system, then this will take up more memory than necessary, since each chunk is 16 MiB.

In practice, the Logstash community also already reverted it back to the original default value of 16MB. elastic/logstash#15765

Modifications

Refer to comment #25021 (comment):

Increasing the default chunk size to 8MB (-Dio.netty.allocator.maxOrder=10) should be something to consider first since that would be the minimal change to mitigate the issue.

In order to maintain compatibility with versions prior to Netty 4.1.76 as much as possible, and to improve chunk utilization (where the chunk size exceeds the default maximum single message size of 5 MB), the Netty configuration -Dio.netty.allocator.maxOrder=10 is added in pulsar.env to set the chunk size to 8 MB.

Verifying this change

• Make sure that the change passes the CI checks.

This change added tests and can be verified as follows:
testDefaultChunkSizeMatchesMaxOrder10

Does this pull request potentially affect one of the following parts:

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete