Account status on on Wavelet node is not working

[asim-sha]

It looks like the account status (balance, stake, reward etc.) is not correctly shown on the Node on both the Status HTTP API as well as the status command on the cmdline. It shows 0 for everything when thats not the case and Lens shows the correct value.

{ public_key: "405c26514497ea02badd3ac5e6e9f9ada353dfa7604ddc418901acdda562ad08", balance: 0, gas_balance: 0, stake: 0, reward: 0, nonce: 0, is_contract: false }

[iwasaki-kenta]

This issue usually comes abrupt if a node that you are personally running is not properly syncing with the testnet. Are there any errors or peers repetitively connecting/disconnecting to/from your node?

[asim-sha]

I will check it again tonight. But issue #268 is probably unrelated because that was happening on lens.perlin.net and not my local node.

[asim-sha]

I've had a look again the does not seem to be in sync. Although its been connected for over an hour it has not synced. I do see this error when starting the node:

�[2m12:08PM�[0m �[1m�[31mERR�[0m�[0m error while checking out of sync with 144.91.73.85:3000 �[31merror: �[0m�[31m"rpc error: code = Unavailable desc = transport is closing"�[0m �[2mevent: �[0msync

And I also see peer repeatedly connecting/disconnecting as you mentioned.

[iwasaki-kenta]

I've had a look again the does not seem to be in sync. Although its been connected for over an hour it has not synced. I do see this error when starting the node:

�[2m12:08PM�[0m �[1m�[31mERR�[0m�[0m error while checking out of sync with 144.91.73.85:3000 �[31merror: �[0m�[31m"rpc error: code = Unavailable desc = transport is closing"�[0m �[2mevent: �[0msync

And I also see peer repeatedly connecting/disconnecting as you mentioned.

Gotcha. The problem then is that your router is blocking nodes in the testnet from connecting to your node. After port-forwarding the port that you assigned your node, you should then be able to properly sync and interact with the testnet.

[asim-sha]

I've had a look again the does not seem to be in sync. Although its been connected for over an hour it has not synced. I do see this error when starting the node:
�[2m12:08PM�[0m �[1m�[31mERR�[0m�[0m error while checking out of sync with 144.91.73.85:3000 �[31merror: �[0m�[31m"rpc error: code = Unavailable desc = transport is closing"�[0m �[2mevent: �[0msync
And I also see peer repeatedly connecting/disconnecting as you mentioned.

Gotcha. The problem then is that your router is blocking nodes in the testnet from connecting to your node. After port-forwarding the port that you assigned your node, you should then be able to properly sync and interact with the testnet.

Opening the port has helped get the account information. But looks like it goes out of sync again very quickly and latest account status returns stale values. It eventually does pick up the changes but takes quite a while (15-30mins). Is this normal?

Also I see a bunch of errors but not sure if they are related:

failed to unmarshal the received message proto: wrong wireType = 2 for field OutOfSync" event: apply 
1:28PM WRN Snowball liveness fault count: 5 last_id: true name: syncer new_id: false
1:28PM WRN Snowball liveness fault count: 0 last_id: false name: syncer new_id: true
1:28PM WRN Snowball liveness fault count: 1 last_id: true name: syncer new_id: false
1:28PM WRN Snowball liveness fault count: 0 last_id: false name: syncer new_id: true
1:28PM ERR error while checking out of sync with 176.223.142.119:3000 error: "rpc error: code = Internal desc = grpc: failed to unmarshal the received message proto: wrong wireType = 2 for field OutOfSync" event: apply
1:28PM WRN Snowball liveness fault count: 5 last_id: true name: syncer new_id: false
1:28PM WRN Snowball liveness fault count: 0 last_id: false name: syncer new_id: true
1:28PM WRN Snowball liveness fault count: 0 last_id: true name: syncer new_id: false
1:28PM WRN Snowball liveness fault count: 0 last_id: false name: syncer new_id: true
1:28PM WRN Snowball liveness fault count: 6 last_id: true name: syncer new_id: false

[asim-sha]

I can also confirm that #268 is an independent issue as its happening regardless of where the transaction was initiated. Its probably a more serious issue than this as Transactions should not be disappearing, its seems like a very critical bug.

I've added more information to #268

[rkeene]

The "Snowball liveness fault" errors are normal warning-level errors that can occur.

The "error while checking out of sync" error likely means you are connecting to a peer running an older version of Wavelet and can safely be ignored as long as it's not too frequent.

It's not normal for the node to become out of sync, but if it does it should detect that it is out of sync with its peers and sync to the latest round and get up-to-date quickly. It will emit messages about this. Do you see any messages similar to:

Noticed that we are out of sync; downloading latest state Snapshot from our peer(s)

?

[asim-sha]

I have just tried again (using himitsu branch). My transactions were not reflecting in the account status on the local node (transactions were in stuck in "received" state but on testnet.perlin.net the tx was successfully"applied" and correct account state was shown).

I also did get the message you have mentioned above:

?[2m6:31PM?[0m ?[32mINF?[0m Noticed that we are out of sync; downloading latest state Snapshot from our peer(s). ?[2mcurrent_round: ?[0m2726033 ?[2mevent: ?[0mapply
?[2m6:31PM?[0m ?[33mDBG?[0m Discovered the round which the majority of our peers are currently in. ?[2mevent: ?[0mapply ?[2mlatest_round: ?[0m2726309 ?[2mlatest_round_root: ?[0mbada114bb7801cae79fc1cdd45eefad3a7ca729dd5caf58087605f995b549630
?[2m6:31PM?[0m ?[33mDBG?[0m Starting up workers to downloaded all chunks of data needed to sync to the latest round... ?[2mevent:
?[0mapply ?[2mnum_chunks: ?[0m37 ?[2mnum_workers: ?[0m16
?[2m6:31PM?[0m ?[33mDBG?[0m Downloaded whatever chunks were available to sync to the latest round, and shutted down all workers. Checking validity of chunks... ?[2mevent: ?[0mapply ?[2mnum_chunks: ?[0m37 ?[2mnum_workers: ?[0m16
?[2m6:31PM?[0m ?[32mINF?[0m All chunks have been successfully verified and re-assembled into a diff. Applying diff... ?[2mevent: ?[0mapply ?[2mnum_chunks: ?[0m37 ?[2mtarget_round: ?[0m2726309
?[2m6:31PM?[0m ?[1m?[31mERR?[0m?[0m Failed to apply re-assembled diff to our ledger state. Restarting sync... ?[31merror: ?[0m?[31m"invalid difference: avl: could not find node a85018ca372e6414ee90fabfccf3ea2d"?[0m ?[2mevent: ?[0mapply ?[2mtarget_round: ?[0m2726309
?[2m6:31PM?[0m ?[33mDBG?[0m Discovered the round which the majority of our peers are currently in. ?[2mevent: ?[0mapply ?[2mlatest_round: ?[0m2726309 ?[2mlatest_round_root: ?[0mbada114bb7801cae79fc1cdd45eefad3a7ca729dd5caf58087605f995b549630
?[2m6:31PM?[0m ?[33mDBG?[0m Starting up workers to downloaded all chunks of data needed to sync to the latest round... ?[2mevent:
?[0mapply ?[2mnum_chunks: ?[0m37 ?[2mnum_workers: ?[0m16
?[2m6:31PM?[0m ?[33mDBG?[0m Downloaded whatever chunks were available to sync to the latest round, and shutted down all workers. Checking validity of chunks... ?[2mevent: ?[0mapply ?[2mnum_chunks: ?[0m37 ?[2mnum_workers: ?[0m16
?[2m6:31PM?[0m ?[32mINF?[0m All chunks have been successfully verified and re-assembled into a diff. Applying diff... ?[2mevent: ?[0mapply ?[2mnum_chunks: ?[0m37 ?[2mtarget_round: ?[0m2726309
?[2m6:31PM?[0m ?[1m?[31mERR?[0m?[0m Failed to apply re-assembled diff to our ledger state. Restarting sync... ?[31merror: ?[0m?[31m"invalid difference: avl: could not find node a85018ca372e6414ee90fabfccf3ea2d"?[0m ?[2mevent: ?[0mapply ?[2mtarget_round: ?[0m2726309

The node never resynced, I am assuming its because of the avl error above.

[asim-sha]

One more thing to note was that even when the node is in sync and there is an active stake it says "...not taking part in consensus"

[rkeene]

The Himitsu branch is an experiment and will not talk to the Testnet meaningfully. Ideally to talk to Testnet you should use v0.2.0.

Can you try again with v0.2.0 and see if you get any of those same errors ?