MDW out of sync

[michele-franchi]

This morning the mdw was out of sync by 13+ keyblocks.
https://mainnet.aeternity.io/mdw/v3/status was returning "mdw_height": 1139188 and "node_height": 1139202.
Need to find out what caused this to avoid similar cases in the future.

[venimus]

2025-06-25 00:18:30.454 [error] ** (Bandit.TransportError) Unrecoverable error: timeout
2025-06-25 02:38:51.043 [error] ** (Bandit.HTTPError) Request line HTTP error: "\r\n"
2025-06-25 02:38:51.450 [error] ** (Plug.Conn.NotSentError) a response was neither set nor sent from the connection
2025-06-25 07:22:09.644 [error] Task #PID<0.8456653.0> started from AeMdw.Sync.Server terminating
** (MatchError) no match of right hand side value: {:error, :not_present}
2025-06-25 07:22:09.645 [info] Sync.Server error: {{:badmatch, {:error, :not_present}}, [{AeMdw.Db.Sync.Contract, :create_contract_create_aetx, 4, [file: ~c"lib/ae_mdw/db/sync/contract.ex", line: 191]}, {AeMdw.Db.Sync.Contract, :"-events_mutations/5-fun-2-", 3, [file: ~c"lib/ae_mdw/db/sync/contract.ex", line: 67]}, {Enum, :"-map/2-lists^map/1-1-", 2, [file: ~c"lib/enum.ex", line: 1703]}, {Enum, :"-map/2-lists^map/1-1-", 2, [file: ~c"lib/enum.ex", line: 1703]}, {AeMdw.Db.Sync.Contract, :events_mutations, 5, [file: ~c"lib/ae_mdw/db/sync/contract.ex", line: 54]}, {AeMdw.Db.Sync.Transaction, :tx_mutations, 1, [file: ~c"lib/ae_mdw/db/sync/transaction.ex", line: 183]}, {AeMdw.Db.Sync.Transaction, :transaction_mutations, 6, [file: ~c"lib/ae_mdw/db/sync/transaction.ex", line: 107]}, {AeMdw.Db.Sync.Block, :"-process_micro_block_mutations/4-fun-2-", 7, [file: ~c"lib/ae_mdw/db/sync/block.ex", line: 195]}]}. Stopping...
2025-06-25 07:47:29.200 [error] ** (Bandit.HTTPError) Request line HTTP error: <<0, 0, 0, 29, 208, 242, 129, 248, 139, 255, 154, 247, 213, 239, 148, 182, 209, 180, 192, 159, 236, 149, 230, 143, 225, 135, 232, 202, 240, 139, 246, 139, 246, 22, 3, 1, 0, 238, 1, 0, 0, 234, 3, 3, 90, 247, 117, 127, 118, 207, ...>>
2025-06-25 08:42:24.942 [error] Task #PID<0.8602997.0> started from AeMdw.Sync.Server terminating
** (MatchError) no match of right hand side value: {:error, :not_present}
2025-06-25 08:42:24.943 [info] Mem Sync.Server error: {{:badmatch, {:error, :not_present}}, [{AeMdw.Db.Sync.Contract, :create_contract_create_aetx, 4, [file: ~c"lib/ae_mdw/db/sync/contract.ex", line: 191]}, {AeMdw.Db.Sync.Contract, :"-events_mutations/5-fun-2-", 3, [file: ~c"lib/ae_mdw/db/sync/contract.ex", line: 67]}, {Enum, :"-map/2-lists^map/1-1-", 2, [file: ~c"lib/enum.ex", line: 1703]}, {Enum, :"-map/2-lists^map/1-1-", 2, [file: ~c"lib/enum.ex", line: 1703]}, {AeMdw.Db.Sync.Contract, :events_mutations, 5, [file: ~c"lib/ae_mdw/db/sync/contract.ex", line: 54]}, {AeMdw.Db.Sync.Transaction, :tx_mutations, 1, [file: ~c"lib/ae_mdw/db/sync/transaction.ex", line: 183]}, {AeMdw.Db.Sync.Transaction, :transaction_mutations, 6, [file: ~c"lib/ae_mdw/db/sync/transaction.ex", line: 107]}, {AeMdw.Db.Sync.Block, :"-process_micro_block_mutations/4-fun-2-", 7, [file: ~c"lib/ae_mdw/db/sync/block.ex", line: 195]}]}
2025-06-25 13:20:16.361 [error] ** (Plug.Conn.NotSentError) a response was neither set nor sent from the connection
2025-06-25 13:20:35.496 [error] ** (Plug.Conn.NotSentError) a response was neither set nor sent from the connection
2025-06-25 13:20:45.399 [error] ** (Plug.Conn.NotSentError) a response was neither set nor sent from the connection
2025-06-25 13:21:01.974 request_id=GExLxGIboFEwXPkBnroj [error] ** (exit) {:aborted, {:function_clause, []}}
2025-06-25 13:21:02.272 request_id=GExLxHPolL-dqIYCwbxC [error] ** (exit) {:aborted, {:function_clause, []}}
2025-06-25 13:21:03.940 [error] Task #PID<0.8940302.0> started from #PID<0.8923251.0> terminating
** (MatchError) no match of right hand side value: {:error, :contract_does_not_exist}
2025-06-25 13:21:03.942 request_id=GExLxNNCUElfrZAAjPCE [error] ** (CaseClauseError) no case clause matching: {:exit, {{:badmatch, {:error, :contract_does_not_exist}}, [{AeMdw.Aex9, :render_account_balance, 3, [file: ~c"lib/ae_mdw/aex9.ex", line: 374]}, {Enum, :"-map/2-lists^map/1-1-", 2, [file: ~c"lib/enum.ex", line: 1703]}, {Enum, :"-map/2-lists^map/1-1-", 2, [file: ~c"lib/enum.ex", line: 1703]}, {AeMdw.Collection, :paginate, 4, [file: ~c"lib/ae_mdw/collection.ex", line: 54]}, {AeMdw.Aex9, :fetch_account_balances, 4, [file: ~c"lib/ae_mdw/aex9.ex", line: 259]}, {Task.Supervised, :invoke_mfa, 2, [file: ~c"lib/task/supervised.ex", line: 101]}, {Task.Supervised, :reply, 4, [file: ~c"lib/task/supervised.ex", line: 36]}]}}
    (ae_mdw 1.104.3) deps/plug/lib/plug/error_handler.ex:80: AeMdwWeb.Router.call/2
2025-06-25 15:44:06.706 [error] ** (Plug.Conn.NotSentError) a response was neither set nor sent from the connection
2025-06-25 16:51:50.642 [error] Task #PID<0.9363729.0> started from AeMdw.Sync.Server terminating
** (MatchError) no match of right hand side value: {:error, :not_present}
2025-06-25 16:51:50.643 [info] Mem Sync.Server error: {{:badmatch, {:error, :not_present}}, [{AeMdw.Db.Sync.Contract, :create_contract_create_aetx, 4, [file: ~c"lib/ae_mdw/db/sync/contract.ex", line: 191]}, {AeMdw.Db.Sync.Contract, :"-events_mutations/5-fun-2-", 3, [file: ~c"lib/ae_mdw/db/sync/contract.ex", line: 67]}, {Enum, :"-map/2-lists^map/1-1-", 2, [file: ~c"lib/enum.ex", line: 1703]}, {Enum, :"-map/2-lists^map/1-1-", 2, [file: ~c"lib/enum.ex", line: 1703]}, {AeMdw.Db.Sync.Contract, :events_mutations, 5, [file: ~c"lib/ae_mdw/db/sync/contract.ex", line: 54]}, {AeMdw.Db.Sync.Transaction, :tx_mutations, 1, [file: ~c"lib/ae_mdw/db/sync/transaction.ex", line: 183]}, {AeMdw.Db.Sync.Transaction, :transaction_mutations, 6, [file: ~c"lib/ae_mdw/db/sync/transaction.ex", line: 107]}, {AeMdw.Db.Sync.Block, :"-process_micro_block_mutations/4-fun-2-", 7, [file: ~c"lib/ae_mdw/db/sync/block.ex", line: 195]}]}
2025-06-25 19:14:26.547 [error] Task #PID<0.9624437.0> started from AeMdw.Sync.Server terminating
** (MatchError) no match of right hand side value: {:error, :not_present}
2025-06-25 19:14:26.548 [info] Mem Sync.Server error: {{:badmatch, {:error, :not_present}}, [{AeMdw.Db.Sync.Contract, :create_contract_create_aetx, 4, [file: ~c"lib/ae_mdw/db/sync/contract.ex", line: 191]}, {AeMdw.Db.Sync.Contract, :"-events_mutations/5-fun-2-", 3, [file: ~c"lib/ae_mdw/db/sync/contract.ex", line: 67]}, {Enum, :"-map/2-lists^map/1-1-", 2, [file: ~c"lib/enum.ex", line: 1703]}, {Enum, :"-map/2-lists^map/1-1-", 2, [file: ~c"lib/enum.ex", line: 1703]}, {AeMdw.Db.Sync.Contract, :events_mutations, 5, [file: ~c"lib/ae_mdw/db/sync/contract.ex", line: 54]}, {AeMdw.Db.Sync.Transaction, :tx_mutations, 1, [file: ~c"lib/ae_mdw/db/sync/transaction.ex", line: 183]}, {AeMdw.Db.Sync.Transaction, :transaction_mutations, 6, [file: ~c"lib/ae_mdw/db/sync/transaction.ex", line: 107]}, {AeMdw.Db.Sync.Block, :"-process_micro_block_mutations/4-fun-2-", 7, [file: ~c"lib/ae_mdw/db/sync/block.ex", line: 195]}]}
2025-06-25 20:14:34.328 [error] Task #PID<0.9730802.0> started from AeMdw.Sync.Server terminating
** (MatchError) no match of right hand side value: {:error, :not_present}
2025-06-25 20:14:34.329 [info] Mem Sync.Server error: {{:badmatch, {:error, :not_present}}, [{AeMdw.Db.Sync.Contract, :create_contract_create_aetx, 4, [file: ~c"lib/ae_mdw/db/sync/contract.ex", line: 191]}, {AeMdw.Db.Sync.Contract, :"-events_mutations/5-fun-2-", 3, [file: ~c"lib/ae_mdw/db/sync/contract.ex", line: 67]}, {Enum, :"-map/2-lists^map/1-1-", 2, [file: ~c"lib/enum.ex", line: 1703]}, {Enum, :"-map/2-lists^map/1-1-", 2, [file: ~c"lib/enum.ex", line: 1703]}, {AeMdw.Db.Sync.Contract, :events_mutations, 5, [file: ~c"lib/ae_mdw/db/sync/contract.ex", line: 54]}, {AeMdw.Db.Sync.Transaction, :tx_mutations, 1, [file: ~c"lib/ae_mdw/db/sync/transaction.ex", line: 183]}, {AeMdw.Db.Sync.Transaction, :transaction_mutations, 6, [file: ~c"lib/ae_mdw/db/sync/transaction.ex", line: 107]}, {AeMdw.Db.Sync.Block, :"-process_micro_block_mutations/4-fun-2-", 7, [file: ~c"lib/ae_mdw/db/sync/block.ex", line: 195]}]}
2025-06-25 20:16:06.780 [error] Task #PID<0.9731652.0> started from AeMdw.Sync.Server terminating
** (MatchError) no match of right hand side value: {:error, :not_present}
2025-06-25 20:16:06.782 [info] Mem Sync.Server error: {{:badmatch, {:error, :not_present}}, [{AeMdw.Db.Sync.Contract, :create_contract_create_aetx, 4, [file: ~c"lib/ae_mdw/db/sync/contract.ex", line: 191]}, {AeMdw.Db.Sync.Contract, :"-events_mutations/5-fun-2-", 3, [file: ~c"lib/ae_mdw/db/sync/contract.ex", line: 67]}, {Enum, :"-map/2-lists^map/1-1-", 2, [file: ~c"lib/enum.ex", line: 1703]}, {Enum, :"-map/2-lists^map/1-1-", 2, [file: ~c"lib/enum.ex", line: 1703]}, {AeMdw.Db.Sync.Contract, :events_mutations, 5, [file: ~c"lib/ae_mdw/db/sync/contract.ex", line: 54]}, {AeMdw.Db.Sync.Transaction, :tx_mutations, 1, [file: ~c"lib/ae_mdw/db/sync/transaction.ex", line: 183]}, {AeMdw.Db.Sync.Transaction, :transaction_mutations, 6, [file: ~c"lib/ae_mdw/db/sync/transaction.ex", line: 107]}, {AeMdw.Db.Sync.Block, :"-process_micro_block_mutations/4-fun-2-", 7, [file: ~c"lib/ae_mdw/db/sync/block.ex", line: 195]}]}
2025-06-25 20:25:57.288 [error] Task #PID<0.9750630.0> started from AeMdw.Sync.Server terminating
** (MatchError) no match of right hand side value: {:error, :not_present}
2025-06-25 20:25:57.289 [info] Sync.Server error: {{:badmatch, {:error, :not_present}}, [{AeMdw.Db.Sync.Contract, :create_contract_create_aetx, 4, [file: ~c"lib/ae_mdw/db/sync/contract.ex", line: 191]}, {AeMdw.Db.Sync.Contract, :"-events_mutations/5-fun-2-", 3, [file: ~c"lib/ae_mdw/db/sync/contract.ex", line: 67]}, {Enum, :"-map/2-lists^map/1-1-", 2, [file: ~c"lib/enum.ex", line: 1703]}, {Enum, :"-map/2-lists^map/1-1-", 2, [file: ~c"lib/enum.ex", line: 1703]}, {AeMdw.Db.Sync.Contract, :events_mutations, 5, [file: ~c"lib/ae_mdw/db/sync/contract.ex", line: 54]}, {AeMdw.Db.Sync.Transaction, :tx_mutations, 1, [file: ~c"lib/ae_mdw/db/sync/transaction.ex", line: 183]}, {AeMdw.Db.Sync.Transaction, :transaction_mutations, 6, [file: ~c"lib/ae_mdw/db/sync/transaction.ex", line: 107]}, {AeMdw.Db.Sync.Block, :"-process_micro_block_mutations/4-fun-2-", 7, [file: ~c"lib/ae_mdw/db/sync/block.ex", line: 195]}]}. Stopping...