Make tracing info from ntx builder available (to note creator)

[mmagician]

After the recent challenges while debugging network accounts (AggLayer), I thought about ways of letting the users know what went wrong with processing their note. The creation of the note may be successful, but its processing might fail for various reasons (e.g. currently it could fail because the note script is previously unseen by the node and it can't fetch it by its root from DataStore, though maybe we can address this particular scenario via this)

Could we somehow make the error trace available to the sender? Maybe this could become accessible via gRPC (it's probably easier to make this accessible to anyone rather than just the sender)?

The main advantage would be that our node team wouldn't have to dig into node logs and debug every failed network tx for the user/dev. This is already painful today, and I'm assuming will become even more so once more devs are onboarded.

[bobbinth]

I wonder if it is possible to make some of the Honeycomb traces public. @Mirko-von-Leipzig - does Honeycomb have such a feature?

If that's possible, we could make traces from the NTX builder available to select users.

[Mirko-von-Leipzig]

Pending and Accepted are already possible in the client as is iiuc. Or more finer grained:

OriginTransactionSubmitted  // User transaction submitted which emits the network note
OriginTransactionCommitted  // User transaction committed
NetworkNoteConsumed         // Network note was consumed in some network transaction
 and committed

what errors would this post? Understanding of course that the note is bundled together with a random set of other notes and could fail for any reason really. And that it fails in multiple different ways across different attempts.

    NetworkNoteStatus::Failed(err) => {
        // Note processing failed unexpectedly
        println!("network note processing failed with error {err}");
    }

[mmagician]

And that it fails in multiple different ways across different attempts.

That's a separate topic of whether this is useful :) but for now we could just default to the first attempt?

Understanding of course that the note is bundled together with a random set of other notes

Hmm right, I forgot about that. But let's assume we would give the user the full error for all notes. Then, they could process the trace however they want, but already then it should be useful enough (i.e. the failed processing of a note gives its error message + note ID) for a power dev to do something about it.

[Mirko-von-Leipzig]

We can probably figure something like that out. The main downside would be:

1. We now need to add a gRPC schema for the network builder to expose this (currently this doesn't have a gRPC interface at all and its completely decoupled from everything else).
2. We need to ensure these errors are sanitized.

And maybe to be sure we're in agreement here - this cannot be a long term solution imo.

[mmagician]

And maybe to be sure we're in agreement here - this cannot be a long term solution imo.

once downsides 1 & 2 (which really are just one-time implementation costs?) are tackled, why not?

[Mirko-von-Leipzig]

(2) is really the main concern I have -- we have to be very certain we're not leaking any internal details. And since network txs include:

• validator IO
• block-producer IO
• store IO

this will also leak those errors and those can really contain anything. Because the underlying assumption we work with is that everything except the RPC API is private.

So while we can probably safely expose the execution error -- except here the DataStore includes reaching out to the store again --, and the proving error, that's all that's realistically safe. This would exclude for example missing note script roots, or missing account IDs from FPI calls to the store. Which is presumably an important thing the users would want to know about.

[igamigo]

Something that may also be a fairly quick win is to emit more verbose logs on execution errors for those that are testing against a local node. Currently, I believe if you want any details about your network transactions, you need to explicitly add a print somewhere in the node binary to output relevant information. If a flag or something similar existed to output more NTX logs (maybe even to a file or something), we can prompt users to try a local node first and see what the outcome is.

[Mirko-von-Leipzig]

That's a good idea. I think we can probably afford (for now) to make it info! so we can see them in production as well.

[igamigo]

Commenting here to document this task's progress: #1770