Expose network note failures

[Mirko-von-Leipzig]

Debugging network notes is painful because these aren't executed locally and therefore developers have very little feedback on what's going wrong.

We could make this easier by storing the note's latest execution error in the database and exposing this via gRPC.

There is a downside to this, which is that the ntx-builder now requires a gRPC server; though at least this also lets us add a status endpoint in the future...

Open questions:

• Do we expose status in general, or only latest error? The latter is much simpler; I would prefer to start there.

cc @mmagician @partylikeits1983 for further inputs.

[SantiagoPittella]

This failing notes should be added to the status? Or to some other endpoint?

I'm thinking that status should only print the overall status of the component, and should not include failing notes. And another endpoint to retrieve notes filtered by a network account in particular. We can store the last N failing notes or something like that.

[Mirko-von-Leipzig]

To a different endpoint e.g. get_note_error(note_id). There are lots of possible options here, I'd like to keep it simple though, so maybe just the exact note ID?

We'll need some guidance from the devx guys though.

[Mirko-von-Leipzig]

Separately, we can also emit an info! for each note failure, as per #1609 (comment). Maybe also get this into main so we can debug sooner.

[SantiagoPittella]

I'll tackle this issue after the actors refactor is merged, to avoid building on top of that branch.

[sergerad]

I think it has been discussed before, but I would imagine this use case would fall under a more general eventing feature rather than being specific to network note errors. Just flagging that we might want to raise that as a discussion if we haven't already.

[Mirko-von-Leipzig]

I think it has been discussed before, but I would imagine this use case would fall under a more general eventing feature rather than being specific to network note errors. Just flagging that we might want to raise that as a discussion if we haven't already.

I don't think we've got a discussion for this; lets create one if you have some specifics in mind? Internally, events should be covered by tracing, so I imagine this would be events for users querying of the non-canonical events? e.g. what state is my transaction in now if it isn't committed - (dropped, in batch, in-mempool etc).

[mmagician]

To a different endpoint e.g. get_note_error(note_id)

Getting the status by node_id should work well enough.

[Mirko-von-Leipzig]

To a different endpoint e.g. get_note_error(note_id)

Getting the status by node_id should work well enough.

We have a status endpoint that returns the status of the component (e.g. RPC, block-producer), which is used to display info on the network monitor. We can't overload this to include note ID queries.

[mmagician]

I think I was overloading the meaning of "status" in my reply: I mean we can get the failure status by node_id, i.e. the endpoint get_note_error(note_id) you suggest should work 👍🏼