terranix: Increase EC2 storage from 50GB to 100GB

[shivaraj-bh]

Postgresql service was failing to start due to low disk space:

Sep 05 11:00:14 chutney postgres[455203]: [455203] FATAL:  could not access status of transaction 0
Sep 05 11:00:14 chutney postgres[455203]: [455203] DETAIL:  Could not write to file "pg_multixact/members/0353" at offset 16384: No space left on device.
Sep 05 11:00:14 chutney postgres[455203]: [455203] CONTEXT:  WAL redo at 79/9F206FD8 for MultiXact/ZERO_MEM_PAGE: 27234
Sep 05 11:00:14 chutney postgres[196210]: [196210] LOG:  startup process (PID 455203) exited with exit code 1
Sep 05 11:00:14 chutney postgres[196210]: [196210] LOG:  terminating any other active server processes
Sep 05 11:00:14 chutney postgres[196210]: [196210] LOG:  shutting down due to startup process failure
Sep 05 11:00:14 chutney postgres[196210]: [196210] LOG:  database system is shut down
Sep 05 11:00:14 chutney systemd[1]: postgresql.service: Main process exited, code=exited, status=1/FAILURE

df -h output:

Filesystem                Size  Used Avail Use% Mounted on
devtmpfs                  390M     0  390M   0% /dev
tmpfs                     3.9G  1.1M  3.9G   1% /dev/shm
tmpfs                     2.0G  7.3M  1.9G   1% /run
/dev/disk/by-label/nixos   49G   44G  3.3G  93% /
efivarfs                  128K  2.6K  126K   3% /sys/firmware/efi/efivars
tmpfs                     1.0M     0  1.0M   0% /run/credentials/systemd-tmpfiles-setup-dev-early.service
tmpfs                     1.0M     0  1.0M   0% /run/credentials/systemd-tmpfiles-setup-dev.service
tmpfs                     3.9G  2.2M  3.9G   1% /run/wrappers
/dev/nvme0n1p1            236M   84M  153M  36% /boot
tmpfs                     1.0M     0  1.0M   0% /run/credentials/systemd-tmpfiles-setup.service
tmpfs                     1.0M     0  1.0M   0% /run/credentials/systemd-journald.service
tmpfs                     780M  4.0K  780M   1% /run/user/0

resolves #46

[srid]

How much data does postgres DB alone take?

[shivaraj-bh]

How much data does postgres DB alone take?

@srid

du -sh /var/lib/postgresql/
25G     /var/lib/postgresql/

(storage used by postgres as of posting this comment)

[srid]

That's more than I'd expected.

1. What does attic actually store in postgres?
2. What is the proportion of DB size growth compared to cache size? Are we going to increase 100G once again in future? If so, when?

[shivaraj-bh]

What does attic actually store in postgres?

• Cache info
• Chunk state
• NAR to chunk binding
• NAR state
• Nix Store object info

What is the proportion of DB size growth compared to cache size?

25GB DB size for 350GB cache objects.

Size graph of the S3 bucket over the last two weeks as a reference:

[shivaraj-bh]

DB size distribution

Looks like the chunk state is taking majority of the space:

Total size and number of rows in each table as a reference:

[shivaraj-bh]

Are we going to increase 100G once again in future? If so, when?

If we keep the same chunking config, we will need (approximately) 74 MB for every 1 GB of chunks.

Click to see the calculation

S3 bucket size (s) = 350GB Database size (d) = 25GB

Database size (approximately) required (in MB) for every 1GB of chunks cached: (d/s)*1024 = (25/350)*1024 =73.143 MB

Above, I have rounded it up to 74MB/1GB of chunks.

From the S3 size growth graph in my comment above: S3 bucket size increased from 211 GB to 342 GB in two weeks.

Estimated database size required for (342-211=131)GB = (131*74)/1024 =9.467 GB. At this rate, we have about 10 weeks (2 weeks * (50GB available/10GB consumed every two weeks)) before we have to increase the storage again.

That doesn’t seem sustainable. I will investigate how beneficial chunking really is (in terms of storage it saves) and tune the chunking parameters accordingly or disable chunking entirely.

Edit: Created #48 to evaluate chunking