Retry logic for transaction receipt fetching.

[humanizersequel]

Sometimes this happens:

Thu 13:35 hypergrid:test.hypr: ERROR provider::util: provider/src/util.rs:713: Failed to fetch transaction receipt: provider=haiku-message-answering-machine, tx_hash=0xe9c9698ee7e81d2500d1c9fd350c110a6f667fbfd13d31e786a5f67b7e8aa93b, source_node=shegridonmyhypetilliuhhh.os, error=RpcTimeout
Thu 13:35 hypergrid:test.hypr: ERROR provider: provider/src/lib.rs:536: Payment validation failed for provider 'haiku-message-answering-machine' from node 'shegridonmyhypetilliuhhh.os': Error fetching transaction receipt for 0xe9c9698ee7e81d2500d1c9fd350c110a6f667fbfd13d31e786a5f67b7e8aa93b: RpcTimeout

The Operator has logic to handle this. The Provider does not. Solution: retry logic for fetching the transaction receipt.

[humanizersequel]

Make sure whatever lands on develop requests and grants timer:distro:sys perms!!!

[humanizersequel]

this retry issue is actually a very bad bug resulting in loss of funds.

Thu 14:03 event loop: process timer:distro:sys doesn't have capability to message process provider:hypergrid:test.hypr
Thu 14:03 eth: handling request from gooberware.os@provider:hypergrid:test.hypr; active_subs len: [1, 2, 2]
Thu 14:03 event loop: process timer:distro:sys doesn't have capability to message process provider:hypergrid:test.hypr
Thu 14:03 eth: handling request from gooberware.os@provider:hypergrid:test.hypr; active_subs len: [1, 2, 2]
Thu 14:03 hypergrid:test.hypr: ERROR provider::util: provider/src/util.rs:713: Failed to fetch transaction receipt: provider=goobersyndicate-testing, tx_hash=0xdcebed45f09fb50f52f4cbb687659194f0f37bc757fcb7964a1a64a70b702eb0, source_node=shegridonmyhypetilliuhhh.os, error=RpcTimeout
Thu 14:03 hypergrid:test.hypr: ERROR provider: provider/src/lib.rs:536: Payment validation failed for provider 'goobersyndicate-testing' from node 'shegridonmyhypetilliuhhh.os': Error fetching transaction receipt for 0xdcebed45f09fb50f52f4cbb687659194f0f37bc757fcb7964a1a64a70b702eb0: RpcTimeout

Need to gib the hypergrid process itself timer perms, and need to implement retry logic on receipt fetch.

[Gohlub]

Now that I think about it, while retry logic will mitigate unstable RPCs from leading to this behavior, we can still encounter situations where unreliable (those with bad RPC access) provider nodes can lead to a loss of funds, right? Maybe we need to include a note that those nodes acting as providers need to have reliable RPC access to mitigate potential future issues.

[humanizersequel]

Yes, but we should just improve the RPC over Hyperware story. This is not an app level problem it should be a built-in.

[Gohlub]

Yeah, funnily enough, I was solving this bug earlier and realized that we actually do in fact have robust fallback system for fetching tx receipts (which relies on the timer module), but because of incorrect perms the system would stop after the first try (successful or not)

[Hallmane]

Fixed in PL PR: hyperware-ai/process_lib#160