Enable brotli compression on hxldash.com

[fititnt]

refs

• Discissions
  • Necessary steps to implement a new country without p-codes (like Brazil) #66 (comment)
  • Necessary steps to implement a new country without p-codes (like Brazil) #66 (comment)
• Apache documentation: https://httpd.apache.org/docs/2.4/mod/mod_brotli.html
• Quick guide on how to enable: https://www.brotli.pro/enable-brotli/apache/

---

Since the response headers of hxldash.com give a hint that the server version is Server: Apache/2.4.29 (Ubuntu) and Apache Module mod_brotli have compatibility of Available in version 2.4.26 and later seems to be a good idea enable brotli compression.

From https://www.brotli.pro/enable-brotli/apache/, the minimum steps would be:

1. run a2enmod brotli as root
2. Add AddOutputFilterByType BROTLI_COMPRESS text/html text/plain text/xml text/css text/javascript application/javascript on the Apache VirtualHost of hxldash.com

One thing we could do is take at least the home page of hxldash.com and one or two dashboards (maybe the common case, and maybe one with lots of data that could be compressed, like the geojsons) and inspect the before/after on the pagesize.

Since is possible to force fake request headers, this could be done after brotli already was enabled.

---

PS.: I can confirm that the https://proxy.hxlstandard.org/data/source does not have brotli enabled (but the response header says server: nginx, so as commented here #66 (comment) it may be less simple to suggest implementation there)

[SimonbJohnson]

This is great research. Thank you. I will update the server this Friday and will enable this!

[SimonbJohnson]

I tried implementing this today, but ran into trouble that the module is not recognised with apache. i tried upgrading apache on a test server and still then it is not recognised.

a2enmod brotli
ERROR: Module brotli does not exist!

I tried implementing the AddOutputFilterByType, but that was not recognised either, so I will investigate more.

Test server is hosted here: http://34.229.180.70/

[fititnt]

Can you return the output of the commands? With the exact OS version I may try a quick install on some VPS just to check this.

# OS Version
cat /etc/*release

# List from where the apache is installed
apt policy apache2

# List available modules, since apache2ctl -M only return enabled
ls /etc/apache2/mods-available/

[fititnt]

@SimonbJohnson @Rydela This comment is a reference for #66 (comment) and this #66 (comment). I will post here on the #85 since is somewhat a way to debug who implement.

---

maybe this is a result of testing the Enable brotli compression on hxldash.com #85 (or just was not checked before, since is very specific), but at this moment Apache compressing HTMLs and JavasScripts, but depending on how the Apache is configured to enable GZip, one of the following conditions are not being used:

1. files with extension .json
2. Content-Type: application/json

Way to test compression: curl strategy

Refs: https://stackoverflow.com/a/18984239/894546

One way to test the content is with compression is using curl --compressed --head https://example.com (-H or --head show just the HTTP headers). Here the example at this very moment:

TL:DR: /static/geoms/topojson/AAA/N/geom.json is not returning Content-Encoding: gzip, but other files are.

# fititnt at fititnt-bravo in ~ [20:22:27]
$ curl --compressed --head https://hxldash.com/static/geoms/topojson/BRA/2/geom.json 
HTTP/1.1 200 OK
Date: Fri, 26 Jun 2020 23:23:06 GMT
Server: Apache/2.4.29 (Ubuntu)
Last-Modified: Fri, 26 Jun 2020 10:36:32 GMT
ETag: "2d498e-5a8fa48559a72"
Accept-Ranges: bytes
Content-Length: 2967950
Content-Type: application/json

# fititnt at fititnt-bravo in ~ [20:23:07]
$ curl --compressed --head https://hxldash.com/static/libs/hxlbites/hxl.js          
HTTP/1.1 200 OK
Date: Fri, 26 Jun 2020 23:23:26 GMT
Server: Apache/2.4.29 (Ubuntu)
Last-Modified: Wed, 20 Feb 2019 12:02:09 GMT
ETag: "cb4a-582522099e320-gzip"
Accept-Ranges: bytes
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 11484
Content-Type: application/javascript

# fititnt at fititnt-bravo in ~ [20:23:30]
$ curl --compressed --head https://hxldash.com/view/341                   
HTTP/1.1 200 OK
Date: Fri, 26 Jun 2020 23:23:44 GMT
Server: Apache/2.4.29 (Ubuntu)
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Type: text/html; charset=utf-8

# fititnt at fititnt-bravo in ~ [20:23:46]
$ curl --version
curl 7.69.1 (x86_64-conda_cos6-linux-gnu) libcurl/7.69.1 OpenSSL/1.1.1g zlib/1.2.11 libssh2/1.9.0
Release-Date: 2020-03-11
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS GSS-API HTTPS-proxy IPv6 Kerberos Largefile libz NTLM NTLM_WB SPNEGO SSL TLS-SRP UnixSockets

Brotly or GZip

From https://daniel.haxx.se/blog/tag/brotli/ (NOVEMBER 29, 2017, curl v7.57.0) says:

Brotli compression
The popular browsers have supported brotli compression method for a while and it has already become widely supported by servers.

Now, curl supports it too and the command line tool’s –compressed option will ask for brotli as well as gzip, if your build supports it. Similarly, libcurl supports it with its CURLOPT_ACCEPT_ENCODING option. The server can then opt to respond using either compression format, depending on what it knows.

According to CertSimple, who ran tests on the top-1000 sites of the Internet, brotli gets contents 14-21% smaller than gzip.

Since depending of the version of the curl as soon as brotli is enabled, the result may be the most efficient, the more specific question from https://stackoverflow.com/a/18984239/894546 instead of use --compressed may be need to debug:

## GZip only
curl -sH 'Accept-encoding: gzip' --head https://hxldash.com/static/geoms/topojson/BRA/2/geom.json
curl -sH 'Accept-encoding: gzip' --head https://hxldash.com/static/libs/hxlbites/hxl.js
curl -sH 'Accept-encoding: gzip' --head https://hxldash.com/view/341  

## Request similar to a browser like Chrome
curl -sH 'Accept-encoding: gzip, deflate, br' --head https://hxldash.com/static/geoms/topojson/BRA/2/geom.json
curl -sH 'Accept-encoding: gzip, deflate, br' --head https://hxldash.com/static/libs/hxlbites/hxl.js
curl -sH 'Accept-encoding: gzip, deflate, br' --head https://hxldash.com/view/341  

Way to test compression: Using Chromium based browsers

Here explain one way https://webmasters.stackexchange.com/a/105061/42038. Requires "Large request rows" option enabled.

This may be good to comprare compressed vs umcompressed. Here a quick print screen, sorted by request size