Speed up database writes with synchronous_commit=off (and full_page_write=off and fsync=off?)

[anthonyfok]

2021-11-22 update: This issue was essentially fixed in PR #105 (Pipeline optimization, Sprint 34–36) merged in June 2021; see [https://github.com//issues/77#issuecomment-975802148](reasons why I am keeping this issue open for the time being]

---

docker-compose up --build currently takes several hours to run. In particular, importing data into PostgreSQL database seems somewhat slow to me, apparently synchronizing to the physical disk for each single operation.

It turns out there are PostgreSQL settings to optimize its speed, and some users report like 10x speed improvement for e.g. CI test runs.

[anthonyfok]

Hi @drotheram, please assign me to this issue (and add labels, link to project, etc.) when you have time. Many thanks!

Tasks:

• Benchmark, before and after. (with time or with docker-compose logs?)
• Relevant fsync=off, synchronous_commit=off, full_page_write=off
• Shutdown postgis/postgis and restart it with just synchronous_commit=off when we are done?
• (optional) Use pg_dump to backup the databases and store them somewhere, probably with a README.md or machine-readable file

Relevant links:

---

Related links (to be sorted):

eatmydata (no, it doesn't work with PostgreSQL

• eatmydata for bash - Google Search

• [MDLSITE-2152] Try out "eatmydata" on ci servers - Moodle Tracker

• Bug #1257036 “services started with eatmydata should remove eatm...” : Bugs : sysvinit package : Ubuntu

• [RFC PATCH] docker: Add support for using eatmydata in the database

• Eat ALL my data or how to LD_PRELOAD with sudo

• postgresql fsync - Google Search

• Don't set fsync=off if you want to keep your data - 2ndQuadrant | PostgreSQL

• synchronous_commit - Google Search

• PostgreSQL: Documentation: 9.4: Asynchronous Commit

• PostgreSQL: Documentation: 9.1: Write Ahead Log

• The "synchronous_commit" parameter

• "synchronous_commit" postgresql docker - Google Search

• Postgres non-durable options Docker container. | No Todo Es Mecánico

• Building a postgres docker image...

• How do I get into a Docker container's shell? - Stack Overflow

• trytond-gis/.drone.yml at develop · tryton/trytond-gis

• postgresql fsync off benchmark - Google Search

• Farewell to fsync(): 10× faster database tests with Docker

• FOSDEM 2019 - PostgreSQL vs. fsync

• A Performance Cheat Sheet for PostgreSQL | Severalnines

• postgresql direct IO - Google Search

• PostgreSQL - hackers - Asynchronous and "direct" IO support for PostgreSQL.

• Fsyncgate: errors on fsync are unrecovarable

• Re: fsync() errors is unsafe and risks data loss [LWN.net]

Misc:

• sdras/awesome-actions: A curated list of awesome actions to use on GitHub