fly postgres backup create/restore is REALLY unstable #308
Description
when we're kicking backup restore with below command
fly postgres backup restore "$DUMP_DB_NAME" --app "$BACKUP_DB" --restore-target-name "$LATEST_BACKUP_ID"
the database won't start, as far as I can see in the logs it's because of some permission error (but sometimes, like 10-20% success rate, it works):
logs
2026-02-27T12:06:36Z app[d892175f40d268] fra [info] INFO Starting clean up.
2026-02-27T12:06:36Z app[d892175f40d268] fra [info] INFO Umounting /dev/vdc from /data
2026-02-27T12:06:37Z app[d892175f40d268] fra [info][ 517.788931] reboot: Restarting system
2026-02-27T12:06:47Z app[d892175f40d268] fra [info]2026-02-27T12:06:47.688631142 [01KJFFCF7RXWGW9F4W9DNMWBYQ:main] Running Firecracker v1.14.1
2026-02-27T12:06:47Z app[d892175f40d268] fra [info]2026-02-27T12:06:47.688840414 [01KJFFCF7RXWGW9F4W9DNMWBYQ:main] Listening on API socket ("/fc.sock").
2026-02-27T12:06:48Z app[d892175f40d268] fra [info] INFO Starting init (commit: 350f2667)...
2026-02-27T12:06:48Z app[d892175f40d268] fra [info] INFO Checking filesystem on /data
2026-02-27T12:06:48Z app[d892175f40d268] fra [info]/dev/vdc: clean, 2534/2621440 files, 3526741/10481664 blocks
2026-02-27T12:06:48Z app[d892175f40d268] fra [info] INFO Mounting /dev/vdc at /data w/ uid: 0, gid: 0 and chmod 0755
2026-02-27T12:06:48Z app[d892175f40d268] fra [info] INFO Resized /data to 42932895744 bytes
2026-02-27T12:06:48Z app[d892175f40d268] fra [info] INFO Preparing to run: `start` as root
2026-02-27T12:06:48Z app[d892175f40d268] fra [info] INFO [fly api proxy] listening at /.fly/api
2026-02-27T12:06:49Z app[d892175f40d268] fra [info]2026/02/27 12:06:49 INFO SSH listening listen_address=[fdaa:2:be18:a7b:61d:91a9:2cef:2]:22
2026-02-27T12:06:49Z app[d892175f40d268] fra [info][INFO] Postgres directory present, ignoring `S3_ARCHIVE_REMOTE_RESTORE_CONFIG`
2026-02-27T12:06:49Z app[d892175f40d268] fra [info]admin | Running...
2026-02-27T12:06:49Z app[d892175f40d268] fra [info]monitor | Running...
2026-02-27T12:06:49Z app[d892175f40d268] fra [info]repmgrd | Running...
2026-02-27T12:06:49Z app[d892175f40d268] fra [info]exporter | Running...
2026-02-27T12:06:49Z runner[d892175f40d268] fra [info]Machine started in 1.714s
2026-02-27T12:06:49Z app[d892175f40d268] fra [info]postgres | Running...
2026-02-27T12:06:49Z app[d892175f40d268] fra [info]proxy | Running...
2026-02-27T12:06:49Z app[d892175f40d268] fra [info]monitor | Waiting for Postgres to be ready...
2026-02-27T12:06:49Z app[d892175f40d268] fra [info]repmgrd | [2026-02-27 12:06:49] [NOTICE] repmgrd (repmgrd 5.4.1) starting up
2026-02-27T12:06:49Z app[d892175f40d268] fra [info]repmgrd | [2026-02-27 12:06:49] [INFO] connecting to database "host=d892175f40d268.vm.nexus-dump-ced348a1-4a4a-4853-89f8-d7c4b056c595.internal port=5433 user=repmgr dbname=repmgr connect_timeout=5"
2026-02-27T12:06:49Z app[d892175f40d268] fra [info]repmgrd | [2026-02-27 12:06:49] [ERROR] connection to database failed
2026-02-27T12:06:49Z app[d892175f40d268] fra [info]repmgrd | [2026-02-27 12:06:49] [DETAIL]
2026-02-27T12:06:49Z app[d892175f40d268] fra [info]repmgrd | connection to server at "d892175f40d268.vm.nexus-dump-ced348a1-4a4a-4853-89f8-d7c4b056c595.internal" (fdaa:2:be18:a7b:61d:91a9:2cef:2), port 5433 failed: Connection refused
2026-02-27T12:06:49Z app[d892175f40d268] fra [info]repmgrd | Is the server running on that host and accepting TCP/IP connections?
2026-02-27T12:06:49Z app[d892175f40d268] fra [info]repmgrd |
2026-02-27T12:06:49Z app[d892175f40d268] fra [info]repmgrd | [2026-02-27 12:06:49] [DETAIL] attempted to connect using:
2026-02-27T12:06:49Z app[d892175f40d268] fra [info]repmgrd | user=repmgr connect_timeout=5 dbname=repmgr host=d892175f40d268.vm.nexus-dump-ced348a1-4a4a-4853-89f8-d7c4b056c595.internal port=5433 fallback_application_name=repmgr options=-csearch_path=
2026-02-27T12:06:49Z app[d892175f40d268] fra [info]repmgrd | exit status 6
2026-02-27T12:06:49Z app[d892175f40d268] fra [info]repmgrd | restarting in 5s [attempt 1]
2026-02-27T12:06:49Z app[d892175f40d268] fra [info]reader error: read ptm: input/output error
2026-02-27T12:06:49Z app[d892175f40d268] fra [info]postgres | 2026-02-27 12:06:49.569 UTC [720] FATAL: data directory "/data/postgresql" has invalid permissions
2026-02-27T12:06:49Z app[d892175f40d268] fra [info]postgres | 2026-02-27 12:06:49.569 UTC [720] DETAIL: Permissions should be u=rwx (0700) or u=rwx,g=rx (0750).
2026-02-27T12:06:49Z app[d892175f40d268] fra [info]proxy | [NOTICE] (721) : New worker (749) forked
2026-02-27T12:06:49Z app[d892175f40d268] fra [info]proxy | [NOTICE] (721) : Loading success.
2026-02-27T12:06:49Z app[d892175f40d268] fra [info]proxy | [WARNING] (749) : bk_db/primary1 changed its IP from (none) to fdaa:2:be18:a7b:61d:91a9:2cef:2 by flydns/dns1.
2026-02-27T12:06:49Z app[d892175f40d268] fra [info]proxy | [WARNING] (749) : Server bk_db/primary1 ('fra.nexus-dump-ced348a1-4a4a-4853-89f8-d7c4b056c595.internal') is UP/READY (resolves again).
2026-02-27T12:06:49Z app[d892175f40d268] fra [info]proxy | [WARNING] (749) : Server bk_db/primary1 administratively READY thanks to valid DNS answer.
2026-02-27T12:06:49Z app[d892175f40d268] fra [info]proxy | [WARNING] (749) : bk_db/pg1 changed its IP from (none) to fdaa:2:be18:a7b:61d:91a9:2cef:2 by flydns/dns1.
2026-02-27T12:06:49Z app[d892175f40d268] fra [info]proxy | [WARNING] (749) : Server bk_db/pg1 ('nexus-dump-ced348a1-4a4a-4853-89f8-d7c4b056c595.internal') is UP/READY (resolves again).
2026-02-27T12:06:49Z app[d892175f40d268] fra [info]proxy | [WARNING] (749) : Server bk_db/pg1 administratively READY thanks to valid DNS answer.
2026-02-27T12:06:49Z app[d892175f40d268] fra [info]postgres | exit status 1
2026-02-27T12:06:49Z app[d892175f40d268] fra [info]postgres | done
2026-02-27T12:06:49Z app[d892175f40d268] fra [info]reader error: read ptm: input/output error
2026-02-27T12:06:50Z app[d892175f40d268] fra [info]failed post-init: failed to establish connection to local node: failed to connect to `host=fdaa:2:be18:a7b:61d:91a9:2cef:2 user=postgres database=postgres`: dial error (dial tcp [fdaa:2:be18:a7b:61d:91a9:2cef:2]:5433: connect: connection refused). Retrying...
2026-02-27T12:06:50Z app[d892175f40d268] fra [info]proxy | [WARNING] (749) : Server bk_db/pg1 is DOWN, reason: Layer7 invalid response, info: "HTTP content check did not match", check duration: 0ms. 1 active and 1 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2026-02-27T12:06:50Z app[d892175f40d268] fra [info]exporter | ERRO[0001] Error opening connection to database (postgresql://flypgadmin:PASSWORD_REMOVED@[fdaa:2:be18:a7b:61d:91a9:2cef:2]:5433/postgres?sslmode=disable): dial tcp [fdaa:2:be18:a7b:61d:91a9:2cef:2]:5433: connect: connection refused source="postgres_exporter.go:1658"
2026-02-27T12:06:51Z app[d892175f40d268] fra [info]failed post-init: failed to establish connection to local node: failed to connect to `host=fdaa:2:be18:a7b:61d:91a9:2cef:2 user=postgres database=postgres`: dial error (dial tcp [fdaa:2:be18:a7b:61d:91a9:2cef:2]:5433: connect: connection refused). Retrying...
2026-02-27T12:06:51Z app[d892175f40d268] fra [info]proxy | [WARNING] (749) : Backup Server bk_db/pg is DOWN, reason: Layer7 invalid response, info: "HTTP content check did not match", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2026-02-27T12:06:51Z app[d892175f40d268] fra [info]proxy | [WARNING] (749) : Server bk_db/primary1 is DOWN, reason: Layer7 invalid response, info: "HTTP content check did not match", check duration: 1ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2026-02-27T12:06:51Z app[d892175f40d268] fra [info]proxy | [ALERT] (749) : backend 'bk_db' has no server available!
2026-02-27T12:06:52Z app[d892175f40d268] fra [info]failed post-init: failed to establish connection to local node: failed to connect to `host=fdaa:2:be18:a7b:61d:91a9:2cef:2 user=postgres database=postgres`: dial error (dial tcp [fdaa:2:be18:a7b:61d:91a9:2cef:2]:5433: connect: connection refused). Retrying...
and also backup creation is really unstable, sometimes it works, sometimes it's not working.
Details
❯ fly postgres backup list -a production-nexus-db
|-----------------|-------|--------|--------------------------|--------------------------|
| ID/NAME | ALIAS | STATUS | END TIME | BEGIN WAL |
|-----------------|-------|--------|--------------------------|--------------------------|
| 20250214T131114 | | FAILED | | 0000000100000266000000AD |
| 20250214T131245 | | FAILED | | 0000000100000266000000AF |
| 20250214T131521 | | FAILED | | 0000000100000266000000B3 |
| 20250214T132025 | | FAILED | | 0000000100000266000000B9 |
| 20250214T132447 | | FAILED | | 0000000100000266000000BF |
| 20251223T164250 | | FAILED | | 0000000300000564000000E2 |
| 20251223T190525 | | FAILED | | 000000030000056500000006 |
| 20251223T191405 | | FAILED | | 000000030000056500000008 |
| 20251230T022033 | | FAILED | | 000000030000056D000000DB |
| 20260107T231859 | | FAILED | | 000000030000057A0000002F |
| 20260208T233309 | | FAILED | | 00000003000005A700000091 |
| 20260208T235042 | | FAILED | | 00000003000005A700000095 |
| 20260209T000429 | | FAILED | | 00000003000005A700000098 |
| 20260209T001835 | | FAILED | | 00000003000005A70000009B |
| 20260209T003246 | | FAILED | | 00000003000005A70000009E |
| 20260209T004710 | | FAILED | | 00000003000005A7000000A1 |
| 20260209T010133 | | FAILED | | 00000003000005A7000000A4 |
| 20260209T011559 | | FAILED | | 00000003000005A7000000A7 |
| 20260209T013035 | | FAILED | | 00000003000005A7000000AB |
| 20260209T014506 | | FAILED | | 00000003000005A7000000AE |
| 20260209T015932 | | FAILED | | 00000003000005A7000000B1 |
| 20260209T031342 | | FAILED | | 00000003000005A7000000CC |
| 20260209T032820 | | FAILED | | 00000003000005A7000000CF |
| 20260209T034310 | | FAILED | | 00000003000005A7000000F8 |
| 20260209T035803 | | FAILED | | 00000003000005A7000000FB |
| 20260209T041249 | | FAILED | | 00000003000005A7000000FE |
| 20260209T042738 | | FAILED | | 00000003000005A800000001 |
| 20260209T044225 | | FAILED | | 00000003000005A800000005 |
| 20260209T045711 | | FAILED | | 00000003000005A800000008 |
| 20260209T051154 | | FAILED | | 00000003000005A80000000B |
| 20260209T052643 | | FAILED | | 00000003000005A80000000E |
| 20260209T054129 | | FAILED | | 00000003000005A800000011 |
| 20260209T065607 | | FAILED | | 00000003000005A800000025 |
| 20260209T071054 | | FAILED | | 00000003000005A800000028 |
| 20260209T072534 | | FAILED | | 00000003000005A80000002B |
| 20260209T074020 | | FAILED | | 00000003000005A80000002E |
| 20260209T075515 | | FAILED | | 00000003000005A800000031 |
| 20260209T081058 | | FAILED | | 00000003000005A800000035 |
| 20260209T082605 | | FAILED | | 00000003000005A800000039 |
| 20260209T084030 | | FAILED | | 00000003000005A80000003C |
| 20260209T085625 | | FAILED | | 00000003000005A800000040 |
| 20260209T091413 | | FAILED | | 00000003000005A800000044 |
| 20260209T092924 | | FAILED | | 00000003000005A800000048 |
| 20260209T104808 | | FAILED | | 00000003000005A800000058 |
| 20260209T110511 | | FAILED | | 00000003000005A80000005C |
| 20260209T112248 | | FAILED | | 00000003000005A800000060 |
| 20260209T113740 | | FAILED | | 00000003000005A800000065 |
| 20260209T115306 | | FAILED | | 00000003000005A800000069 |
| 20260209T121053 | | FAILED | | 00000003000005A80000006D |
| 20260209T122745 | | FAILED | | 00000003000005A800000071 |
| 20260209T124325 | | FAILED | | 00000003000005A800000075 |
| 20260209T125952 | | FAILED | | 00000003000005A800000079 |
| 20260209T131731 | | FAILED | | 00000003000005A80000007D |
| 20260209T133205 | | FAILED | | 00000003000005A800000080 |
| 20260209T144729 | | FAILED | | 00000003000005A800000090 |
| 20260209T150317 | | FAILED | | 00000003000005A800000094 |
| 20260209T151845 | | FAILED | | 00000003000005A800000098 |
| 20260209T153604 | | FAILED | | 00000003000005A80000009C |
| 20260209T155207 | | FAILED | | 00000003000005A8000000A0 |
| 20260209T160843 | | FAILED | | 00000003000005A8000000A4 |
| 20260209T162450 | | FAILED | | 00000003000005A8000000A8 |
| 20260209T163932 | | FAILED | | 00000003000005A8000000AB |
| 20260209T165709 | | FAILED | | 00000003000005A8000000AF |
| 20260209T171133 | | FAILED | | 00000003000005A8000000B2 |
| 20260209T172605 | | FAILED | | 00000003000005A8000000B5 |
| 20260209T184027 | | FAILED | | 00000003000005A8000000C4 |
| 20260209T185450 | | FAILED | | 00000003000005A8000000C7 |
| 20260209T190915 | | FAILED | | 00000003000005A8000000CA |
| 20260209T192337 | | FAILED | | 00000003000005A8000000CD |
| 20260209T193804 | | FAILED | | 00000003000005A8000000D0 |
| 20260209T195234 | | FAILED | | 00000003000005A8000000D3 |
| 20260209T200722 | | FAILED | | 00000003000005A8000000D6 |
| 20260209T202156 | | FAILED | | 00000003000005A8000000D9 |
| 20260209T203620 | | FAILED | | 00000003000005A8000000DC |
| 20260209T205051 | | FAILED | | 00000003000005A8000000DF |
| 20260209T210518 | | FAILED | | 00000003000005A8000000E2 |
| 20260209T221944 | | FAILED | | 00000003000005A8000000F1 |
| 20260209T223405 | | FAILED | | 00000003000005A8000000F4 |
| 20260209T224825 | | FAILED | | 00000003000005A8000000F7 |
| 20260209T230244 | | FAILED | | 00000003000005A8000000FA |
| 20260209T232019 | | FAILED | | 00000003000005A8000000FE |
| 20260209T233412 | | FAILED | | 00000003000005A900000001 |
| 20260209T234824 | | FAILED | | 00000003000005A900000004 |
| 20260210T000240 | | FAILED | | 00000003000005A900000007 |
| 20260210T001658 | | FAILED | | 00000003000005A90000000A |
| 20260210T003121 | | FAILED | | 00000003000005A90000000D |
| 20260210T004545 | | FAILED | | 00000003000005A900000010 |
| 20260210T015942 | | FAILED | | 00000003000005A90000001F |
| 20260210T021403 | | FAILED | | 00000003000005A900000022 |
| 20260210T022827 | | FAILED | | 00000003000005A900000025 |
| 20260210T024247 | | FAILED | | 00000003000005A900000028 |
| 20260210T025709 | | FAILED | | 00000003000005A90000002B |
| 20260210T031158 | | FAILED | | 00000003000005A90000003B |
| 20260210T032628 | | FAILED | | 00000003000005A90000003E |
| 20260210T034051 | | FAILED | | 00000003000005A900000041 |
| 20260210T035511 | | FAILED | | 00000003000005A900000044 |
| 20260210T040931 | | FAILED | | 00000003000005A900000047 |
| 20260210T042350 | | FAILED | | 00000003000005A90000004A |
| 20260210T053746 | | FAILED | | 00000003000005A900000059 |
| 20260210T055208 | | FAILED | | 00000003000005A90000005C |
| 20260210T060632 | | FAILED | | 00000003000005A90000005F |
| 20260210T062055 | | FAILED | | 00000003000005A900000062 |
| 20260210T063523 | | FAILED | | 00000003000005A900000065 |
| 20260210T065239 | | FAILED | | 00000003000005A90000006E |
| 20260210T070656 | | FAILED | | 00000003000005A900000071 |
| 20260210T072139 | | FAILED | | 00000003000005A900000074 |
| 20260210T073643 | | FAILED | | 00000003000005A900000078 |
| 20260210T075318 | | FAILED | | 00000003000005A90000007C |
| 20260210T080817 | | FAILED | | 00000003000005A90000007F |
| 20260210T092325 | | FAILED | | 00000003000005A90000008F |
| 20260210T094029 | | FAILED | | 00000003000005A900000093 |
| 20260210T095609 | | FAILED | | 00000003000005A900000097 |
| 20260210T101338 | | FAILED | | 00000003000005A90000009B |
| 20260210T103041 | | FAILED | | 00000003000005A90000009F |
| 20260210T104822 | | FAILED | | 00000003000005A9000000A3 |
| 20260210T110328 | | FAILED | | 00000003000005A9000000A7 |
| 20260210T111948 | | FAILED | | 00000003000005A9000000AB |
| 20260210T113534 | | FAILED | | 00000003000005A9000000AF |
| 20260210T115051 | | FAILED | | 00000003000005A9000000B3 |
| 20260210T120721 | | FAILED | | 00000003000005A9000000B7 |
| 20260210T132221 | | FAILED | | 00000003000005A9000000C8 |
| 20260210T133910 | | FAILED | | 00000003000005A9000000CC |
| 20260210T135704 | | FAILED | | 00000003000005A9000000D0 |
| 20260210T141605 | | FAILED | | 00000003000005A9000000D4 |
| 20260210T143218 | | FAILED | | 00000003000005A9000000D8 |
| 20260210T144956 | | FAILED | | 00000003000005A9000000DC |
| 20260210T150424 | | FAILED | | 00000003000005A9000000DF |
| 20260210T151944 | | FAILED | | 00000003000005A9000000E4 |
| 20260210T153518 | | FAILED | | 00000003000005A9000000E8 |
| 20260210T154713 | | FAILED | | 00000003000005A9000000EB |
| 20260210T155835 | | FAILED | | 00000003000005A9000000EE |
| 20260210T163811 | | FAILED | | 00000003000005A9000000F9 |
| 20260210T165224 | | FAILED | | 00000003000005A9000000FC |
| 20260210T170351 | | FAILED | | |
| 20260210T170629 | | FAILED | | |
| 20260210T173232 | | FAILED | | 00000004000005AA0000000C |
| 20260210T173519 | | FAILED | | 00000004000005AA0000000C |
| 20260210T174441 | | FAILED | | 00000004000005AA0000000F |
| 20260210T174759 | | FAILED | | 00000004000005AA00000010 |
| 20260210T180140 | | FAILED | | 00000004000005AA00000013 |
| 20260210T181538 | | FAILED | | 00000004000005AA00000017 |
| 20260210T182845 | | FAILED | | 00000004000005AA0000001A |
| 20260210T184029 | | FAILED | | 00000004000005AA0000001D |
| 20260210T185214 | | FAILED | | 00000004000005AA00000020 |
| 20260210T190327 | | FAILED | | 00000004000005AA00000023 |
| 20260210T191540 | | FAILED | | 00000004000005AA00000026 |
| 20260210T192659 | | FAILED | | 00000004000005AA00000029 |
| 20260210T193816 | | FAILED | | 00000004000005AA0000002C |
| 20260210T204943 | | FAILED | | 00000004000005AA0000003B |
| 20260210T210145 | | FAILED | | 00000004000005AA0000003E |
| 20260210T211326 | | FAILED | | 00000004000005AA00000041 |
| 20260210T212437 | | FAILED | | 00000004000005AA00000044 |
| 20260210T213631 | | FAILED | | 00000004000005AA00000047 |
| 20260210T214813 | | FAILED | | 00000004000005AA0000004A |
| 20260210T215953 | | FAILED | | 00000004000005AA0000004D |
| 20260210T221134 | | FAILED | | 00000004000005AA00000050 |
| 20260210T222317 | | FAILED | | 00000004000005AA00000053 |
| 20260210T223503 | | FAILED | | 00000004000005AA00000056 |
| 20260210T224646 | | FAILED | | 00000004000005AA00000059 |
| 20260210T235802 | | FAILED | | 00000004000005AA00000068 |
| 20260211T000954 | | FAILED | | 00000004000005AA0000007B |
| 20260211T002821 | | FAILED | | 00000004000005AA00000088 |
| 20260211T004516 | | FAILED | | 00000004000005AA0000008C |
| 20260211T010044 | | FAILED | | 00000004000005AA00000092 |
| 20260211T011240 | | FAILED | | 00000004000005AA00000095 |
| 20260211T012439 | | FAILED | | 00000004000005AA00000098 |
| 20260211T013551 | | FAILED | | 00000004000005AA0000009B |
| 20260211T014701 | | FAILED | | 00000004000005AA0000009E |
| 20260211T015914 | | FAILED | | 00000004000005AA000000A1 |
| 20260211T021036 | | FAILED | | 00000004000005AA000000A5 |
| 20260211T032150 | | FAILED | | 00000004000005AA000000C1 |
| 20260211T033347 | | FAILED | | 00000004000005AA000000C4 |
| 20260211T034528 | | FAILED | | 00000004000005AA000000C7 |
| 20260211T035710 | | FAILED | | 00000004000005AA000000CA |
| 20260211T040856 | | FAILED | | 00000004000005AA000000CD |
| 20260211T042040 | | FAILED | | 00000004000005AA000000D0 |
| 20260211T043223 | | FAILED | | 00000004000005AA000000D3 |
| 20260211T044404 | | FAILED | | 00000004000005AA000000D6 |
| 20260211T045544 | | FAILED | | 00000004000005AA000000D9 |
| 20260211T050652 | | FAILED | | 00000004000005AA000000DC |
| 20260211T051843 | | FAILED | | 00000004000005AA000000DF |
| 20260211T063002 | | FAILED | | 00000004000005AA000000EE |
| 20260211T064147 | | FAILED | | 00000004000005AA000000F6 |
| 20260211T065744 | | FAILED | | 00000004000005AA000000FA |
| 20260211T071015 | | FAILED | | 00000004000005AA000000FD |
| 20260211T072129 | | FAILED | | 00000004000005AB00000000 |
| 20260211T073407 | | FAILED | | 00000004000005AB00000003 |
| 20260211T074645 | | FAILED | | 00000004000005AB00000006 |
| 20260211T075923 | | FAILED | | 00000004000005AB00000009 |
| 20260211T081304 | | FAILED | | 00000004000005AB0000000C |
| 20260211T082625 | | FAILED | | 00000004000005AB0000000F |
| 20260211T084009 | | FAILED | | 00000004000005AB00000012 |
| 20260211T095434 | | FAILED | | 00000004000005AB00000022 |
| 20260211T102059 | | FAILED | | 00000004000005AB00000028 |
| 20260211T103100 | | FAILED | | 00000004000005AB0000002B |
| 20260211T104144 | | FAILED | | 00000004000005AB0000002F |
| 20260211T105121 | | FAILED | | 00000004000005AB00000031 |
| 20260211T110101 | | FAILED | | 00000004000005AB00000033 |
| 20260211T111926 | | FAILED | | 00000004000005AB00000038 |
| 20260211T112157 | | FAILED | | 00000004000005AB00000039 |
| 20260211T112544 | | FAILED | | 00000004000005AB0000003A |
| 20260211T112949 | | FAILED | | 00000004000005AB0000003B |
| 20260211T123434 | | FAILED | | 00000004000005AB00000048 |
| 20260211T124110 | | FAILED | | 00000004000005AB0000004A |
| 20260211T124804 | | FAILED | | 00000004000005AB0000004C |
| 20260211T130439 | | FAILED | | 00000004000005AB00000052 |
| 20260211T130704 | | FAILED | | 00000004000005AB00000053 |
| 20260211T131047 | | FAILED | | 00000004000005AB00000054 |
| 20260211T131236 | | FAILED | | 00000004000005AB00000055 |
| 20260211T131407 | | FAILED | | 00000004000005AB00000056 |
| 20260212T160518 | | FAILED | | 00000004000005AC000000E7 |
| 20260220T042638 | | DONE | Fri Feb 20 04:48:29 2026 | 00000004000005B30000001D |
| 20260220T054843 | | DONE | Fri Feb 20 06:10:19 2026 | 00000004000005B300000024 |
| 20260220T071048 | | DONE | Fri Feb 20 07:33:33 2026 | 00000004000005B300000033 |
| 20260220T083409 | | DONE | Fri Feb 20 09:00:17 2026 | 00000004000005B300000040 |
| 20260220T100050 | | DONE | Fri Feb 20 10:29:16 2026 | 00000004000005B30000004E |
| 20260220T112947 | | DONE | Fri Feb 20 11:54:19 2026 | 00000004000005B30000005B |
| 20260220T125452 | | DONE | Fri Feb 20 13:20:22 2026 | 00000004000005B300000067 |
| 20260220T142045 | | DONE | Fri Feb 20 14:43:36 2026 | 00000004000005B300000072 |
| 20260220T154404 | | DONE | Fri Feb 20 16:08:47 2026 | 00000004000005B30000007D |
| 20260220T170925 | | DONE | Fri Feb 20 17:33:41 2026 | 00000004000005B300000086 |
| 20260220T183412 | | DONE | Fri Feb 20 18:57:16 2026 | 00000004000005B30000008E |
| 20260220T195746 | | DONE | Fri Feb 20 20:20:32 2026 | 00000004000005B300000095 |
| 20260220T212105 | | DONE | Fri Feb 20 21:43:31 2026 | 00000004000005B30000009E |
| 20260220T224400 | | DONE | Fri Feb 20 23:06:38 2026 | 00000004000005B3000000A5 |
| 20260221T000713 | | DONE | Sat Feb 21 00:29:32 2026 | 00000004000005B3000000AC |
| 20260221T013005 | | DONE | Sat Feb 21 01:52:31 2026 | 00000004000005B3000000B3 |
| 20260221T025301 | | DONE | Sat Feb 21 03:15:26 2026 | 00000004000005B3000000BA |
| 20260221T041602 | | DONE | Sat Feb 21 04:38:37 2026 | 00000004000005B3000000D0 |
| 20260221T053856 | | DONE | Sat Feb 21 06:01:29 2026 | 00000004000005B3000000D7 |
| 20260221T070200 | | DONE | Sat Feb 21 07:24:54 2026 | 00000004000005B3000000E5 |
| 20260221T082525 | | DONE | Sat Feb 21 08:48:44 2026 | 00000004000005B3000000ED |
| 20260221T094920 | | DONE | Sat Feb 21 10:12:42 2026 | 00000004000005B3000000F5 |
| 20260221T111315 | | DONE | Sat Feb 21 11:36:38 2026 | 00000004000005B3000000FD |
| 20260221T123754 | | DONE | Sat Feb 21 13:01:36 2026 | 00000004000005B400000005 |
| 20260221T140211 | | DONE | Sat Feb 21 14:25:57 2026 | 00000004000005B40000000D |
| 20260221T152629 | | DONE | Sat Feb 21 15:50:10 2026 | 00000004000005B400000015 |
| 20260221T165042 | | DONE | Sat Feb 21 17:14:46 2026 | 00000004000005B40000001D |
| 20260221T181517 | | DONE | Sat Feb 21 18:39:08 2026 | 00000004000005B400000025 |
| 20260221T193945 | | DONE | Sat Feb 21 20:03:41 2026 | 00000004000005B40000002D |
| 20260221T210415 | | DONE | Sat Feb 21 21:28:14 2026 | 00000004000005B400000035 |
| 20260221T222848 | | DONE | Sat Feb 21 22:52:45 2026 | 00000004000005B40000003D |
| 20260221T235319 | | DONE | Sun Feb 22 00:17:23 2026 | 00000004000005B40000007D |
| 20260222T011759 | | DONE | Sun Feb 22 01:41:58 2026 | 00000004000005B400000085 |
| 20260222T024232 | | DONE | Sun Feb 22 03:06:36 2026 | 00000004000005B40000008D |
| 20260222T040710 | | DONE | Sun Feb 22 04:31:06 2026 | 00000004000005B4000000A5 |
| 20260222T053143 | | DONE | Sun Feb 22 05:55:42 2026 | 00000004000005B4000000AC |
| 20260222T065611 | | DONE | Sun Feb 22 07:20:23 2026 | 00000004000005B4000000BA |
| 20260222T082057 | | DONE | Sun Feb 22 08:45:11 2026 | 00000004000005B4000000C1 |
| 20260222T094542 | | DONE | Sun Feb 22 10:09:58 2026 | 00000004000005B4000000CA |
| 20260222T111028 | | DONE | Sun Feb 22 11:35:03 2026 | 00000004000005B4000000D1 |
| 20260222T123535 | | DONE | Sun Feb 22 13:00:30 2026 | 00000004000005B4000000D8 |
| 20260222T140103 | | DONE | Sun Feb 22 14:25:17 2026 | 00000004000005B4000000DF |
| 20260222T152549 | | DONE | Sun Feb 22 15:50:23 2026 | 00000004000005B4000000E7 |
| 20260222T165058 | | DONE | Sun Feb 22 17:16:12 2026 | 00000004000005B4000000EE |
| 20260222T181641 | | DONE | Sun Feb 22 18:41:02 2026 | 00000004000005B4000000F6 |
| 20260222T194134 | | DONE | Sun Feb 22 20:06:23 2026 | 00000004000005B4000000FD |
| 20260222T210656 | | DONE | Sun Feb 22 21:32:10 2026 | 00000004000005B500000005 |
| 20260222T223238 | | DONE | Sun Feb 22 22:57:24 2026 | 00000004000005B50000000C |
| 20260222T235755 | | DONE | Mon Feb 23 00:22:43 2026 | 00000004000005B500000013 |
| 20260223T012314 | | DONE | Mon Feb 23 01:47:44 2026 | 00000004000005B50000001B |
| 20260223T024815 | | DONE | Mon Feb 23 03:14:03 2026 | 00000004000005B500000023 |
| 20260223T041434 | | DONE | Mon Feb 23 04:39:34 2026 | 00000004000005B50000003D |
| 20260223T053953 | | DONE | Mon Feb 23 06:05:04 2026 | 00000004000005B500000046 |
| 20260223T070533 | | DONE | Mon Feb 23 07:30:39 2026 | 00000004000005B500000054 |
| 20260223T083110 | | DONE | Mon Feb 23 08:58:43 2026 | 00000004000005B50000005D |
| 20260223T095916 | | DONE | Mon Feb 23 10:27:32 2026 | 00000004000005B50000006D |
| 20260223T112804 | | DONE | Mon Feb 23 11:53:49 2026 | 00000004000005B50000007B |
| 20260223T125419 | | DONE | Mon Feb 23 13:23:35 2026 | 00000004000005B500000087 |
| 20260223T142355 | | DONE | Mon Feb 23 14:49:37 2026 | 00000004000005B500000093 |
| 20260223T155012 | | FAILED | | 00000004000005B50000009E |
| 20260223T160125 | | DONE | Mon Feb 23 16:27:21 2026 | 00000004000005B5000000A0 |
| 20260223T172752 | | DONE | Mon Feb 23 17:52:22 2026 | 00000004000005B5000000A9 |
| 20260223T185253 | | DONE | Mon Feb 23 19:18:24 2026 | 00000004000005B5000000B1 |
| 20260223T201856 | | DONE | Mon Feb 23 20:43:52 2026 | 00000004000005B5000000B9 |
| 20260223T214424 | | DONE | Mon Feb 23 22:10:37 2026 | 00000004000005B5000000C0 |
| 20260223T231110 | | DONE | Mon Feb 23 23:37:09 2026 | 00000004000005B5000000C8 |
| 20260224T003741 | | DONE | Tue Feb 24 01:03:31 2026 | 00000004000005B5000000CF |
| 20260224T020404 | | DONE | Tue Feb 24 02:30:08 2026 | 00000004000005B5000000D6 |
| 20260224T033043 | | DONE | Tue Feb 24 03:56:33 2026 | 00000004000005B5000000EE |
| 20260224T045714 | | DONE | Tue Feb 24 05:23:05 2026 | 00000004000005B5000000F5 |
| 20260224T062337 | | DONE | Tue Feb 24 06:49:45 2026 | 00000004000005B5000000FD |
| 20260224T075015 | | DONE | Tue Feb 24 08:17:44 2026 | 00000004000005B60000000B |
| 20260224T091814 | | DONE | Tue Feb 24 09:46:12 2026 | 00000004000005B600000016 |
| 20260224T104643 | | DONE | Tue Feb 24 11:14:10 2026 | 00000004000005B600000023 |
| 20260224T121445 | | DONE | Tue Feb 24 12:41:32 2026 | 00000004000005B600000031 |
| 20260224T134200 | | DONE | Tue Feb 24 14:09:37 2026 | 00000004000005B60000003E |
| 20260224T151011 | | DONE | Tue Feb 24 15:38:15 2026 | 00000004000005B60000004B |
| 20260224T163847 | | DONE | Tue Feb 24 17:04:43 2026 | 00000004000005B600000055 |
| 20260224T180514 | | DONE | Tue Feb 24 18:30:37 2026 | 00000004000005B60000005E |
| 20260224T193112 | | DONE | Tue Feb 24 19:56:59 2026 | 00000004000005B600000066 |
| 20260224T205729 | | DONE | Tue Feb 24 21:23:56 2026 | 00000004000005B60000006D |
| 20260224T222425 | | DONE | Tue Feb 24 22:50:43 2026 | 00000004000005B600000074 |
| 20260224T235113 | | DONE | Wed Feb 25 00:17:22 2026 | 00000004000005B60000007C |
| 20260225T011753 | | DONE | Wed Feb 25 01:43:39 2026 | 00000004000005B6000000A7 |
| 20260225T024409 | | DONE | Wed Feb 25 03:10:31 2026 | 00000004000005B6000000AF |
| 20260225T041102 | | DONE | Wed Feb 25 04:36:57 2026 | 00000004000005B6000000C7 |
| 20260225T053732 | | DONE | Wed Feb 25 06:03:26 2026 | 00000004000005B6000000CE |
| 20260225T070400 | | DONE | Wed Feb 25 07:30:36 2026 | 00000004000005B6000000DC |
| 20260225T083111 | | DONE | Wed Feb 25 08:58:58 2026 | 00000004000005B6000000E6 |
| 20260225T095928 | | DONE | Wed Feb 25 10:27:42 2026 | 00000004000005B6000000F5 |
| 20260225T112817 | | DONE | Wed Feb 25 11:57:57 2026 | 00000004000005B700000001 |
| 20260225T125829 | | DONE | Wed Feb 25 13:26:47 2026 | 00000004000005B70000000D |
| 20260225T142718 | | DONE | Wed Feb 25 14:54:57 2026 | 00000004000005B70000001A |
| 20260225T155529 | | DONE | Wed Feb 25 16:22:26 2026 | 00000004000005B700000026 |
| 20260225T172257 | | DONE | Wed Feb 25 17:49:29 2026 | 00000004000005B700000030 |
| 20260225T185003 | | DONE | Wed Feb 25 19:16:25 2026 | 00000004000005B700000039 |
| 20260225T201656 | | DONE | Wed Feb 25 20:43:29 2026 | 00000004000005B700000041 |
| 20260225T214358 | | DONE | Wed Feb 25 22:10:26 2026 | 00000004000005B700000048 |
| 20260225T231056 | | DONE | Wed Feb 25 23:38:28 2026 | 00000004000005B700000051 |
| 20260226T003857 | | DONE | Thu Feb 26 01:10:29 2026 | 00000004000005B70000005D |
| 20260226T021100 | | DONE | Thu Feb 26 02:38:02 2026 | 00000004000005B700000069 |
| 20260226T033831 | | DONE | Thu Feb 26 04:05:10 2026 | 00000004000005B700000080 |
| 20260226T050544 | | DONE | Thu Feb 26 05:32:31 2026 | 00000004000005B700000087 |
| 20260226T063257 | | DONE | Thu Feb 26 07:01:55 2026 | 00000004000005B70000008F |
| 20260226T080224 | | DONE | Thu Feb 26 08:30:40 2026 | 00000004000005B7000000A0 |
| 20260226T093116 | | DONE | Thu Feb 26 09:59:41 2026 | 00000004000005B7000000AB |
| 20260226T110016 | | DONE | Thu Feb 26 11:28:16 2026 | 00000004000005B7000000B7 |
| 20260226T122850 | | DONE | Thu Feb 26 12:56:43 2026 | 00000004000005B7000000C2 |
| 20260226T135719 | | DONE | Thu Feb 26 14:25:39 2026 | 00000004000005B7000000D0 |
| 20260226T152610 | | DONE | Thu Feb 26 15:55:17 2026 | 00000004000005B7000000DC |
| 20260226T165552 | | DONE | Thu Feb 26 17:24:08 2026 | 00000004000005B7000000E8 |
| 20260226T182441 | | DONE | Thu Feb 26 18:52:21 2026 | 00000004000005B7000000F1 |
| 20260226T195257 | | DONE | Thu Feb 26 20:20:37 2026 | 00000004000005B7000000F8 |
| 20260226T212105 | | FAILED | | 00000004000005B800000000 |
| 20260226T213932 | | DONE | Thu Feb 26 22:07:25 2026 | 00000004000005B800000002 |
| 20260226T230756 | | DONE | Thu Feb 26 23:37:25 2026 | 00000004000005B80000000B |
| 20260227T003801 | | DONE | Fri Feb 27 01:05:50 2026 | 00000004000005B800000013 |
| 20260227T020625 | | DONE | Fri Feb 27 02:34:09 2026 | 00000004000005B80000001B |
| 20260227T033442 | | DONE | Fri Feb 27 04:02:26 2026 | 00000004000005B800000033 |
| 20260227T050256 | | DONE | Fri Feb 27 05:30:46 2026 | 00000004000005B80000003B |
| 20260227T063119 | | DONE | Fri Feb 27 06:59:40 2026 | 00000004000005B800000043 |
| 20260227T080016 | | DONE | Fri Feb 27 08:29:31 2026 | 00000004000005B800000052 |
| 20260227T093004 | | DONE | Fri Feb 27 09:59:34 2026 | 00000004000005B80000005E |
| 20260227T110008 | | DONE | Fri Feb 27 11:33:20 2026 | 00000004000005B8000000AA |
|-----------------|-------|--------|--------------------------|--------------------------|
this is our db app toml file:
# fly.toml app configuration file generated for production-nexus-db on 2026-02-27T13:12:10+01:00
#
# See https://fly.io/docs/reference/configuration/ for information about how to use this file.
#
app = 'production-nexus-db'
primary_region = 'fra'
[env]
FLY_RESTORED_FROM = 'vol_4y7w3lnj7n7ejqj4'
PRIMARY_REGION = 'fra'
[[mounts]]
source = 'pg_data'
destination = '/data'
[[services]]
protocol = 'tcp'
internal_port = 5432
auto_start_machines = false
[[services.ports]]
port = 5432
handlers = ['pg_tls']
[services.concurrency]
type = 'connections'
hard_limit = 1000
soft_limit = 1000
[[services]]
protocol = 'tcp'
internal_port = 5433
auto_start_machines = false
[[services.ports]]
port = 5433
handlers = ['pg_tls']
[services.concurrency]
type = 'connections'
hard_limit = 1000
soft_limit = 1000
[checks]
[checks.pg]
port = 5500
type = 'http'
interval = '15s'
timeout = '10s'
path = '/flycheck/pg'
[checks.role]
port = 5500
type = 'http'
interval = '15s'
timeout = '10s'
path = '/flycheck/role'
[checks.vm]
port = 5500
type = 'http'
interval = '15s'
timeout = '10s'
path = '/flycheck/vm'
[[metrics]]
port = 9187
path = '/metrics'