Implement TX kernel support for faucet callbacks

[PhilippGackstatter]

This issue is about implementing the tx kernel support for asset callbacks, i.e. the ability for faucets to implement blocklists or allowlists and future similar functionality.

Callback Storage

How should callbacks be stored on-chain? This was explored in #2328 (comment), but this issue refines that approach some more.

The basis of the proposal is still to have a list of well-known callbacks and assign each of them an index, for instance:

• 0 -> procedure called when an asset is added to an account's vault, i.e. on_asset_added_to_account.
• 1 -> procedure called when an asset is added to a note, i.e. on_asset_added_to_note.

Storage Map

This is the approach from #2328 (comment), which proposes to store callbacks in a storage list (#2176) in the faucet like this:

[
  // on_asset_added_to_account
  0x0000000000000000000000000000000000000000000000000000000000000000
  // on_asset_added_to_note
  0x497690421c9bb611b124c489557dbd65a8729db802a86199500749ec97b0342d
]

This works overall, but has the disadvantage that storage lists or maps require an extra call from the client to the node to fetch the witness for it.

Value Storage Slot Approach

This approach avoids the extra call from the client to the node with the following overall approach based on @bobbinth's idea: Callback index 0 (e.g. "on_asset_added_to_account") is assigned to 8 bits in a value storage slot. These 8 bits are interpreted as the index of an account procedure in the faucet's account code that is the callback. This means two levels of indirection.

In this approach, we standardize one value-storage slot at the protocol level, for example miden::protocol::faucet::callbacks.

It consists of four felts where the 8 least significant bits of felt 0 are callback index 0, the next 8 bits of felt 0 are callback index 1, and so on. To visualize the layout of this word:

0: [ zero bit | enabled_callbacks (7 bits) | idx=6  | idx=5  | idx=4  | idx=3  | idx=2  | idx=1  | idx=0  ]
1: [ zero bit | enabled_callbacks (7 bits) | idx=13 | idx=12 | idx=11 | idx=10 | idx=9  | idx=8  | idx=7  ]
2: [ zero bit | enabled_callbacks (7 bits) | idx=20 | idx=19 | idx=18 | idx=17 | idx=16 | idx=15 | idx=14 ]
3: [ zero bit | enabled_callbacks (7 bits) | idx=27 | idx=26 | idx=25 | idx=24 | idx=23 | idx=22 | idx=21 ]

The most significant bit is set to zero to guarantee it is a valid felt, and so this makes the most significant byte unusable as an index. So, one value storage slot has enough room for 28 callback indices. If we need more, we can add miden::protocol::asset::callbacks2 in the future.

Each index is 8 bits, which means it can point to any of the 256 possible account procedures. This is good because it means we don't rely on account code having a certain layout, e.g. auth procedure at index 0, "on_asset_added_to_account" callback at index 1, whether account procedures are sorted or unsorted by MAST root, etc.

Each callback index has exactly 8 bits, which is exactly the max possible number of procedures in an account. Unlike the map approach, we cannot set a root to the empty word to signal that the callback isn't enabled and shouldn't be called by the kernel (for example to signal that a faucet does not implement on_asset_added_to_account).

Instead, the remaining 7 bits of the most significant byte can be used for this purpose. These enabled_callbacks bits determine whether the callback at the corresponding bit index is enabled or not. For instance, if 0b0001_0010 is the most significant byte of element 2 in the storage slot, then callbacks with indices 15 and 18 are enabled, the remaining callbacks in that felt aren't.

In terms of upgradability, it would be easy to enforce that if the total number of standardized callbacks is currently 7, then elements 1, 2 and 3 of the storage slot need to be 0. That way, if we increase the number of standardized callbacks to 8, accounts will not have that callback unintentionally enabled.

Migrating from on_asset_added_to_account = idx 0 to on_asset_added_to_account_v2 = idx 10 can be done by setting the appropriate bits to 0 and 1, respectively.

Since value slots are part of storage slot headers, the callbacks are loaded into memory when the foreign account is loaded, and no further lazy loading of data should be necessary.

The downside of this approach are the two levels of indirection, which make this conceptually more complicated. Since this can be abstracted away from the user behind nicer abstractions, this is not really an issue that affects users. Secondly, it will require quite some bit manipulation in the kernel, but that should also not be a big issue as long as it is well tested.

Integrated Account Code Approach

I also considered whether we can represent the callback indices somehow as the indices of the procedures in account code. For example, "on_asset_added_to_account" must be at index 1, "on_asset_added_to_note" must be at index 2, etc. Disabled callbacks can be set to empty words, which would lead to some otherwise unnecessary padding in the account code. It is problematic that if we add a new callback at index 3 later, the account code will likely have a non-empty MAST root stored at this location, and so the callback would be unintentionally enabled. Hence, I don't think this is a good approach.

Conclusion

My favorite option would now be the value storage slot approach.

Callback Interfaces

We also need to define what the procedure signature of each callback looks like. For now, this only deals with two callbacks and more can be added later.

Asset Added To Account

#! Inputs:  [account_id, ASSET_KEY, ASSET_VALUE]
#! Outputs: [PROCESSED_ASSET_KEY, PROCESSED_ASSET_VALUE]
#!
#! Where:
#! - account_id is the ID of the account to whose vault the asset is being added.
#! - ASSET* is the asset that is being added.
#! - PROCESSED_ASSET* is the asset after being processed by the callback. It is the asset that will
#!   eventually be added to the account's vault.
pub proc on_asset_added_to_account

The processed asset probably isn't very useful for now as I believe it must always be identical to the passed-in asset. That's because the foreign account has no way of splitting a part of the asset off and moving it to a note, or minting additional tokens, etc. If it doesn't return the original asset, it would violate asset preservation rules. I assume we'll want faucets to have this ability eventually, so it may make sense to already add it. We can also start simpler - either approach shouldn't matter much for the implementation complexity.

The main open question is whether a reference to the account's vault should also be passed, as mentioned in approach 3. I think this would be a bit of a break to our existing pattern, where foreign account's do not get access to the native account's data, except for its account ID, and access is by calling ("pull") instead of data being passed to it ("push"). E.g. the native account ID can also be accessed by calling miden::protocol::native_account::get_id. Calling any other native account procedure would currently fail with ERR_FOREIGN_ACCOUNT_CONTEXT_AGAINST_NATIVE_ACCOUNT. If we do want to make the vault accessible to the foreign account, the established pattern would be to make it accessible through a kernel procedure, e.g. native_account::get_asset, or by allowing foreign accounts to call native account procedures (tbd in terms of security).

This would give the native account the control of whether to have a procedure in its interface that allows or disallows access in more controlled ways, e.g. checking whether an account owner is over 18 by calling a procedure of the appropriate standardized "identity interface", instead of granting access to the entire vault unconditionally.

Asset Added To Note

#! Inputs:  [note_idx, ASSET_KEY, ASSET_VALUE]
#! Outputs: [PROCESSED_ASSET_KEY, PROCESSED_ASSET_VALUE]
#!
#! Where:
#! - note_idx is the index of the note to which the asset is being added.
#! - ASSET* is the asset that is being added.
#! - PROCESSED_ASSET* is the asset after being processed by the callback. It is the asset that will
#!   eventually be added to the note's assets.
pub proc on_asset_added_to_note

The note index is useful because the foreign account can call miden::protocol::output_note::get_assets and some other getters to inspect the other assets of the note. Later it may also be useful to add data to the note's attachment as discussed elsewhere (currently restricted to when active account = native account).

I'm not as deep into the requirements of these callbacks, so I appreciate any input on this cc @onurinanc @MCarlomagno.

Standard Storage Slot

We'll probably also want at least a "standard storage slot" (like TokenMetadata) that makes this callback storage slot nicely accessible. I'm not sure yet how faucet owners would mutate the callbacks storage slot after initial creation, so an account component that allows for mutation may not be necessary right away. Instead, a standard storage FaucetCallbacks slot may be enough for now. But I haven't thought about this a lot.

Other Thoughts

• One of the bigger follow-ups from the callback workstream more broadly will probably be thinking about what APIs we can allow foreign accounts to safely access. For instance, faucet::{mint, burn} are currently restricted to the native account, but it should be safe to allow faucet A to mint more of its own assets during a callback, since it can define its own logic.
• merge, split, compare (see Expand asset layout from one to two words #2328) and other such asset procedures aren't really callbacks, they are rather the "asset interface". So putting them into the same slot as the callbacks may be most efficient, but doesn't feel right on a conceptual level. Moving them to a separate storage slot may be fine, too.

[bobbinth]

Thank you for writing this up! This largely makes sense. A few questions/comments below:

This works overall, but has the disadvantage that storage lists or maps require an extra call from the client to the node to fetch the witness for it.

Would this still be true if we go with the StorageList approach described in #2176 (comment)? Basically, I'm wondering if we implement storage lists first, would we still need to have this level of indirection?

Each callback index has exactly 8 bits, which is exactly the max possible number of procedures in an account. Unlike the map approach, we cannot set a root to the empty word to signal that the callback isn't enabled and shouldn't be called by the kernel (for example to signal that a faucet does not implement on_asset_added_to_account).

Could we not say that value 0 implies that the callback is not enabled? My thinking here is that procedure at index 0 is the auth procedure and we can probably safely assume that auth procedures are not going to be used for callbacks. So, we should be able to use index == 0 as a sentinel value to indicate that a callback is not enabled.

We also need to define what the procedure signature of each callback looks like. For now, this only deals with two callbacks and more can be added later.

If I'm understanding correctly, on_asset_added_to_account and on_asset_added_to_note would be assigned the first two "slots" in miden::protocol::faucet::callbacks word (i.e., bits 0 - 16) - right?

Then, the logic to figure out if, for example, we want to call on_asset_added_to_account would work as follows:

1. When moving an asset into an account, make an FPI call to the faucet to read the miden::protocol::faucet::callbacks storage slot (we can get the faucet ID from the ASSET_KEY).
2. Extract bits 0 - 8 from the storage slot.
3. If the bits are set to 0, we don't need to make the callback. But if the bits are set to a non-zero value, find a procedure at that index, and invoke it with the parameters defined by the on_asset_added_to_account procedure interface.

If this is correct, I do wonder if we can avoid the first step in some cases. Specifically, to read the miden::protocol::faucet::callbacks of the faucet, we would need to make the call to the node to get the account header. It will be just a single call (since we don't need to download any storage maps), but it is one call nevertheless. And it would be great to avoid this call for tokens that don't require any callbacks.

I think somewhere previously we discussed using the 8 "free" bits we have in the ASSET_KEY (because faucet ID is only 120 bits) to encode some extra info about callbacks into the asset itself. I think there are roughly 3 ways to go about it:

1. We could use the 8 bits just to encode a binary value - e.g., 0 means we don't need to make any callbacks, and 1 means we need to make some callbacks. Then, 1 would trigger getting the account header of the faucet from the node to determine exactly which callbacks we want to make.
2. We could use the 8 bits to encode "callback families". For example: 0 means no callbacks, 1 means we have only the on_asset_added_to_account callback, 2 means we have only the on_asset_added_to_note call back, 3 means we have both callbacks etc.
3. We could use the 8 bits as a bitfield. for example, 0000_0000 means no callbacks; 0000_0001 means on_asset_added_to_account is enabled, 0000_0010 means on_asset_added_to_note is enabled; 0000_0011 means both callbacks are enabled (numerically this looks the same as the case above, but that's a coincidence). With this approach we could also have named storage slots for each callback - e.g., we could have miden::protocol::faucet::callbacks::on_asset_added_to_account slot which would store the root of the procedure to be called when an asset is added to the account.

I kind of like approach 3 because it avoids the need to decode the miden::protocol::faucet::callbacks and then look up procedures by index. The main limitation here is we could have at most 8 callbacks - but maybe that's enough?

Also, this could be a marker to identify "native" assets. That is, for native assets, the top 6 bits of the bitfield would be set to zeros. Though, maybe we'll have other ways of identifying native assets.

The processed asset probably isn't very useful for now as I believe it must always be identical to the passed-in asset. That's because the foreign account has no way of splitting a part of the asset off and moving it to a note, or minting additional tokens, etc. If it doesn't return the original asset, it would violate asset preservation rules. I assume we'll want faucets to have this ability eventually, so it may make sense to already add it. We can also start simpler - either approach shouldn't matter much for the implementation complexity.

I think having the on_asset_added_to_account callback return PROCESSED_ASSET is fine but a couple points,

• It is not clear to me that we should return both the key and the value. My understanding was that they key should remain static in call cases (i.e., the faucet should not be able to manipulate the asset key).
• In the future, we may be able to use it for merge procedure as well as the faucet could read the account's vault and modify the value of the added asset accordingly - though, maybe this confuses things too much.

It is not yet clear to me yet whether on_asset_added_to_note should also be able to manipulate the asset value (but maybe this doesn't hurt).

The main open question is whether a reference to the account's vault should also be passed, as mentioned in approach 3. I think this would be a bit of a break to our existing pattern, where foreign account's do not get access to the native account's data, except for its account ID, and access is by calling ("pull") instead of data being passed to it ("push"). E.g. the native account ID can also be accessed by calling miden::protocol::native_account::get_id. Calling any other native account procedure would currently fail with ERR_FOREIGN_ACCOUNT_CONTEXT_AGAINST_NATIVE_ACCOUNT. If we do want to make the vault accessible to the foreign account, the established pattern would be to make it accessible through a kernel procedure, e.g. native_account::get_asset, or by allowing foreign accounts to call native account procedures (tbd in terms of security).

Agreed. I think the way to go here is probably to expose native_account::get_asset to be accessible in the FPI context.

---

To summarize my main points/questions:

• I think it would be great not to request faucet account headers from the node for faucets that don't require any callbacks, and maybe for this we could use the 8 "free" bits in the ASSET_KEY.
• Would implementing Add support for multi-value storage slots #2176 first affect our design somehow (if we have dedicated storage slot per callback, we probably don't care).
• I'm wondering how we'd differentiate between native and non-native assets in the future. It may be worth to think about this even now.
• The callbacks shouldn't probably have the ability to modify the ASSET_KEY.
• We don't need to pass vault references into the callbacks.

[PhilippGackstatter]

Would this still be true if we go with the StorageList approach described in #2176 (comment)? Basically, I'm wondering if we implement storage lists first, would we still need to have this level of indirection?

If we used a storage list that is basically part of the account' storage header, then I don't think this extra call to the node would be needed. So the list is a viable approach. If we go with the list, I think it still makes sense to pack 7 callbacks in one felt, and so one word in a list would also be able to hold 28 callback indices. So, I think using a list with that approach makes sense if we anticipate needing a lot more than 28 callbacks, which doesn't look that likely. We may never have more than 28 callbacks, so a single slot does seem simpler.

Could we not say that value 0 implies that the callback is not enabled? My thinking here is that procedure at index 0 is the auth procedure and we can probably safely assume that auth procedures are not going to be used for callbacks.

Yes, assuming we will not need sorting for account procedure roots (which would mean having a header word at the beginning of the account procedure section that contains the index of the auth procedure). This depends on how we implement account code upgrades in the account delta (#2183).

For now, we can implement this by interpreting 0 as "unset" and change it later, if needed.

Specifically, to read the miden::protocol::faucet::callbacks of the faucet, we would need to make the call to the node to get the account header.
And it would be great to avoid this call for tokens that don't require any callbacks.
I think somewhere previously we discussed using the 8 "free" bits we have in the ASSET_KEY (because faucet ID is only 120 bits) to encode some extra info about callbacks into the asset itself.

That would be great, I agree. I wrote down some ideas for using these 8 bits in #2328 (comment). To summarize, I thought we would only need a callback enabled/disabled flag for built-in assets, because for custom assets, we would always need to call merge or split in order to add/remove that asset from the account. Therefore, to get merge/split, we would need to fetch the callback storage slot anyway and so a callback flag would not be useful because we could just check the storage header directly.

I think that assumption may not necessarily be true: Say we're adding some custom asset A to an account's vault and we check the existing value in the vault and it returns Word::empty, meaning no asset exists. Then we can interpret this state in two ways:

1. Either call issuer::merge(Word::empty(), A_value),
2. Or assume that Word::empty is the identity element, and merge(Word::empty(), A_value) = A_value for all types of assets. Therefore, no call to merge is actually necessary, and so we don't even need to fetch the issuer account's storage header or code.

My assumption was 1, but I think 2 is actually necessary. For non-fungible assets, merge/split will not be defined and calling it would panic, so I think we have to use the logic from 2. This would consequently save us from these calls if the asset we're adding is non-fungible, does not exist in the vault (no merge necessary), or is removed entirely (no split necessary). I think this will happen not that often, but at least some of the time.

Long story, short: I mention this because the downstream consequence is that a callback enabled/disabled flag probably makes sense for built-in and custom assets, which matches your assumption.

I kind of like approach 3 because it avoids the need to decode the miden::protocol::faucet::callbacks and then look up procedures by index. The main limitation here is we could have at most 8 callbacks - but maybe that's enough?

I like the idea in principle but 8 callbacks sounds a bit restrictive. I think we'll need at least 5 callbacks in the short to mid-term (merge, split, compare, on_asset_added_to_account, on_asset_added_to_note). This leaves room for only 3 others before we'd have to come up with another mechanism.

It's also not as efficient as packing multiple indices into a single storage slot. With the
miden::protocol::faucet::callbacks approach we can have up to 28 callbacks encoded into a single slot whereas the bitfield approach would require 28 storage slots to encode the same data.

Separately, assuming we do not need or want accounts to be able to issue multiple assets, then we can indeed use the 8 bits freely. I would basically use your first option: 0 means callbacks are globally disabled for this faucet and 1 means callbacks are globally enabled - meaning the tx kernel fetches that faucet's storage header to check more precisely. So, individual enabled/disabled flags for each concrete callback still exist in in the storage header somewhere (by checking if the procedure index is 0, for instance).

Also, this could be a marker to identify "native" assets. That is, for native assets, the top 6 bits of the bitfield would be set to zeros. Though, maybe we'll have other ways of identifying native assets.

My favorite thing would be to not have native/built-in assets at all 🙂. On a technical level, we can do this by generalizing assets and abstracting the construction of the fee asset away into a user-defined "fee script/procedure" that runs after the auth procedure. We basically pass that script the "gas"/"computation units" used by the transaction, and it returns the asset to be used as the fee. That would also allow for paying fees in any asset and as a side effect massively simplify the tx kernel and allow us to move our current fungible/non-fungible assets to miden::standards.

It is not clear to me that we should return both the key and the value. My understanding was that they key should remain static in call cases (i.e., the faucet should not be able to manipulate the asset key).

Agreed, that was an oversight.

In the future, we may be able to use it for merge procedure as well as the faucet could read the account's vault and modify the value of the added asset accordingly - though, maybe this confuses things too much.

I'd probably favor keeping them separate to keep it simpler.

To summarize:

• I think it would be great not to request faucet account headers from the node for faucets that don't require any callbacks, and maybe for this we could use the 8 "free" bits in the ASSET_KEY.

• I agree it makes sense to optimize this with a callback enabled/disabled flag.
• I would use a single bitflag in the asset key that tells the kernel whether it should fetch the storage header of the faucet or not and have individual callback flags encoded in storage to give users the ability to enable/disable every callback as needed.

• Would implementing Add support for multi-value storage slots #2176 first affect our design somehow (if we have dedicated storage slot per callback, we probably don't care).

Unless the single-storage slot bit encoding I proposed is too complicated or inefficient in MASM, I would prefer that over encoding MAST roots or indices in a storage list, mainly to keep things more packed. I think a single storage slot with 28 callbacks may be sufficient for the forseeable future.

• I'm wondering how we'd differentiate between native and non-native assets in the future. It may be worth to think about this even now.

If we need that distinction, we can dedicate 1 or 2 bits in the asset key to that, but my favorite option is very much to get rid of native and non-native assets before mainnet.

• The callbacks shouldn't probably have the ability to modify the ASSET_KEY.
• We don't need to pass vault references into the callbacks.

Agreed.

[bobbinth]

I would basically use your first option: 0 means callbacks are globally disabled for this faucet and 1 means callbacks are globally enabled - meaning the tx kernel fetches that faucet's storage header to check more precisely.

Yes, I think this should work.

My favorite thing would be to not have native/built-in assets at all

I think we do need to be able to identify the "asset category" somehow. Conceptually, I think the asset key would need to encode the following enum:

enum AssetCategory {
    /// The kernel knows how to merge/split the asset and no callbacks are needed
    Native,
    /// The kernel knows how to merge/split the asset but needs to make callbacks to the faucet
    NativeWithCallbacks,
    /// The kernel relies on the faucet for merge/split logic
    Custom,
}

I'm imagining this info would be used in account::add_asset_to_vault procedure roughly as follows (and other relevant procedures could work similarly):

#! Inputs:  [ASSET_KEY, ASSET_VALUE]
#! Outputs: [ASSET_VALUE']
pub proc add_asset_to_vault
    ...
    
    # extract asset category from ASSET_KEY
    # => [asset_category, ASSET_KEY, ASSET_VALUE]

    # for now, assert that the category is either Native or NativeWithCallbacks
    # => [has_callbacks, ASSET_KEY, ASSET_VALUE]

    if.true
        # invoke on_asset_added_to_account for the faucet
    end
    # => [ASSET_KEY, PROCESSED_ASSET_VALUE]
    ...

    exec.asset_vault::add_asset
    # => [FINAL_ASSET_VALUE]

    ...
end

I think one question here is how do we actually invoke the on_asset_added_to_account on the faucet - ideally, we'd be able to re-use as much of FPI infrastructure for this as possible.

One way to go about this is to have the procedures defined via dynamic execution. For example, on_asset_added_to_account on call faucets could look something like:

const ON_ASSET_ADDED_PROC_ROOT=word("miden::protocol::faucet::callbacks::on_asset_added_to_account")

#! Inputs:  [account_id, ASSET_KEY, ASSET_VALUE]
#! Outputs: [PROCESSED_ASSET_VALUE]
pub proc on_asset_added_to_account
    push.ON_ASSET_ADDED_PROC_ROOT[0..2]
    exec.active::account::get_item
    # => [PROC_ROOT, account_id, ASSET_KEY, ASSET_VALUE]

    # maybe here we can check if PROC_ROOT == EMPTY_WORD and if it is
    # skip executing the code below

    loc_storew.0 dropw
    locaddr.0
    => [proc_ptr, account_id, ASSET_KEY, ASSET_VALUE]

    # invoke the procedure
    dynexec
end

The above does require one storage slot per callback, but I think it should simplify their handling.

1. Either call issuer::merge(Word::empty(), A_value),
2. Or assume that Word::empty is the identity element, and merge(Word::empty(), A_value) = A_value for all types of assets. Therefore, no call to merge is actually necessary, and so we don't even need to fetch the issuer account's storage header or code.

My assumption was 1, but I think 2 is actually necessary. For non-fungible assets, merge/split will not be defined and calling it would panic, so I think we have to use the logic from 2.

I've also assumed it would be 1 - and I'm not sure I understand why that wouldn't work. The non-fungible assets could still have merge/ split procedures defined - but these procedures would basically be no ops.

I like the idea in principle but 8 callbacks sounds a bit restrictive.

Agreed - though, I do think that 28 callback budget is probably an overkill. I would maybe even go with the word encoding a procedure index per element to simplify things (assuming the simplification is meaningful).

---

To summarize:

• I would use the 8 unused bits in the ASSET_KEY to encode "asset category". For now, we have two categories: "native" and "native with callbacks". In the future, we'll add the custom category (or maybe even more than one custom category).
• For the "native with callbacks" category, we can use the scheme you've suggested for encoding the actual callbacks. Though, I think we can also assume that this category has only two callbacks (the on_asset_added_to_account and on_asset_added_to_note). The extension mechanism here would be to add more categories which could have a different set of callbacks in the future. In any case, I'd prioritize simplicity over storage efficiency.

[PhilippGackstatter]

Conceptually, I think the asset key would need to encode the following enum:

enum AssetCategory {
/// The kernel knows how to merge/split the asset and no callbacks are needed
Native,
/// The kernel knows how to merge/split the asset but needs to make callbacks to the faucet
NativeWithCallbacks,
/// The kernel relies on the faucet for merge/split logic
Custom,
}

One thing I realized: When encoding this into the unused 8 bits, it allows faucets to mint the same asset once with callbacks enabled and once without. That's because the kernel cannot know whether previous assets were minted with or without that flag and so cannot enforce that to be the same. So, a USDC faucet could mint USDC(50, callback=enabled) and also USDC(50, callback=disabled). Because the asset keys would be different, these assets would be considered completely distinct, which is probably bad (the wallet would end up showing you have 2 instances of 50 USDC which is highly confusing, since one would expect 1 instance of 100 USDC).

Options:

• Accept this situation and leave it up to faucets to not do the wrong thing. I can't think of a way to use this is a feature, so it basically leaves the door open for a bug / is a footgun.
• Encode the callback flag into the account ID to make it immutable. Works, but puts asset-related things into the account ID, which I don't like conceptually.
• Ignore the callback flag when using asset keys in the vault, specifically: Before comparing asset keys, before inserting an asset or before retrieving an asset from the vault, zero out the callback bit to effectively make assets of the same category equal, independent of the callback flag. We'd have to be diligent to do this in all the right places and we can add asset::is_key_equal to implement this comparison.
• Require faucets to have a protocol-reserved, immutable storage slot that contains the callback flag. It seems unnecessary to have this be a storage slot because it is immutable, so there "should" be a different option.

I would conceptually prefer the "ignore callback flag" option. With proper test coverage, I think we could make sure the kernel implements this correctly. I'm mainly worried that users will end up comparing keys manually with word::eq, because we'd be communicating both "assets with the same key are identical" on the one hand but also "you have to zero this one bit in the asset key before comparing", and so there's quite a bit of subtlety. So (unfortunately) the account ID option sounds safest to me.

• The extension mechanism here would be to add more categories which could have a different set of callbacks in the future.

That makes sense!

If I understand correctly, you'd go with the one callback per storage slot approach. Concretely, we would add two protocol-defined storage slots now that are:

• miden::protocol::faucet::callbacks::on_asset_added_to_account contains PROC_ROOT.
• miden::protocol::faucet::callbacks::on_asset_added_to_note contains PROC_ROOT.

If the slot does not exist or the slot contains Word::empty, then the callback will not be invoked.

If you'd prefer simplicity over storage efficiency, then that's a good option. The callback-index-per-felt approach is also reasonable, so I can just see which of these two flavors is easier to implement or handle.

I've also assumed it would be 1 - and I'm not sure I understand why that wouldn't work. The non-fungible assets could still have merge/ split procedures defined - but these procedures would basically be no ops.

True, option 1 can work if we change the "interface" that users need to implement:

• With option 1, the interface that the Rust compiler could define is merge(A: Option<MyAsset>, B: MyAsset) in order to handle the case where the existing asset A is not present / the default value. Here Word::empty would be mapped to None by the compiler wrapper.
  • We can also enforce that Word::empty must be a valid instance of every asset, but I don't think it's a good idea to force users to do that.
• With option 2, users need to implement merge(A: MyAsset, B: MyAsset), i.e. users can assume that A and B are valid instances of the asset.
  • The caveat here is that Word::empty is automatically assumed to either not be a valid instance of the asset or the default representation of the asset (e.g. fungible asset with amount 0). I think this is a better option than enforcing that Word::empty must be a valid encoding of the asset type.

With option 2, the tx kernel could take care of unwrapping the Options before invoking the custom merge/split procedures, roughly like this:

fn merge(value0: Word, value1: Word) {
    if value0.is_empty() {
        return value1;
    }

    if value1.is_empty() {
        return value0;
    }

    // We only need to fetch the faucet's state from chain if we get here.
    faucet::merge(value0, value1)
}

Option 1 means we'd have to always fetch the merge/split logic from faucets (API call to the node), while option 2 does not necessarily require that, which is a small advantage.

Option 1 is more flexible, which can be an advantage, though I also think it's an advantage if the Rust compiler can offer a nice interface like merge(MyAsset, MyAsset) instead of only merge(Word, Word) where users need to take care of all possible cases themselves.

But that's not necessarily something we need to figure out now.

[bobbinth]

One thing I realized: When encoding this into the unused 8 bits, it allows faucets to mint the same asset once with callbacks enabled and once without. That's because the kernel cannot know whether previous assets were minted with or without that flag and so cannot enforce that to be the same. So, a USDC faucet could mint USDC(50, callback=enabled) and also USDC(50, callback=disabled). Because the asset keys would be different, these assets would be considered completely distinct, which is probably bad (the wallet would end up showing you have 2 instances of 50 USDC which is highly confusing, since one would expect 1 instance of 100 USDC).

• Accept this situation and leave it up to faucets to not do the wrong thing. I can't think of a way to use this is a feature, so it basically leaves the door open for a bug / is a footgun.

I'd probably go with this option. Assets that have callbacks vs. assets that don't are in my mind fundamentally different assets (e.g., one can be frozen and the other one cannot) - and so should not be "fungible" with each other (e.g., adding 5 units of "freezable" asset to 5 units of "non-freezable" asset shouldn't result in 10 units of one or the other).

I do agree that is is a footgun of sorts, but I'd say we have many similar footguns which we don't try to prevent at the protocol level (e.g., not enforcing NFT uniqueness). So, I think it is fine to leave this in the realm of faucet developer responsibility.

An alternative would be to encode the flat into the account ID - but I agree with you that it would be better to move faucet-related info into the ASSET_KEY rather than the other way around.

This actually makes me think: should we move fungible/non-fungible faucet info into the asset key rather than keep it in the account ID? Basically, I think we can reduce AccountType enum to be something like:

pub enum AccountType {
    Account, // could also be SimpleAccount or RegularAccount
    Faucet,
}

And then the code in the faucet would determine what kind of assets the faucet can issue. I think something like this would be required to support custom assets anyways (as these we won't be able to classify as fungible or non-fungible).

This does mean that we'd need to use 2 bits out of the 8 unused bits to encode info about "built-in" assets. Basically, the AssetCategory enum would look something like this:

enum AssetCategory {
    NativeFungible,
    NativeFungibleWithCallbacks,
    NativeNonFungible,
    NativeNoFungibleWithCallbacks,
    Custom,
}

If I understand correctly, you'd go with the one callback per storage slot approach. Concretely, we would add two protocol-defined storage slots now that are:

• miden::protocol::faucet::callbacks::on_asset_added_to_account contains PROC_ROOT.
• miden::protocol::faucet::callbacks::on_asset_added_to_note contains PROC_ROOT.

If the slot does not exist or the slot contains Word::empty, then the callback will not be invoked.

Yes, that's what I had in mind but a couple of comments:

• To make things simpler, I thought these storage slots could always be present in the account, and if the callback is not needed, the relevant slot would be set to an empty word.
• Also to make things simpler, the kernel could always invoke the wrapper procedure, and inside the wrapper procedure we'd read the storage slot and would decide whether to invoke the actual callback procedure dynamically or not.

[PhilippGackstatter]

Assets that have callbacks vs. assets that don't are in my mind fundamentally different assets

As discussed yesterday, that also works 👍

To make things simpler, I thought these storage slots could always be present in the account, and if the callback is not needed, the relevant slot would be set to an empty word.

Sounds good as well.

Also to make things simpler, the kernel could always invoke the wrapper procedure, and inside the wrapper procedure we'd read the storage slot and would decide whether to invoke the actual callback procedure dynamically or not.

I agree this structure should be simple and easy to follow.

This actually makes me think: should we move fungible/non-fungible faucet info into the asset key rather than keep it in the account ID?

Agreed, though what is the reason we need AccountType at all then? Since we no longer have protocol-reserved slots, the only distinction between the simple and faucet account type would be that one can issue assets and the other can't. But similar to account code mutability/immutability, this could also be decided at the account component level, i.e. accounts that have faucet components will be able to issue assets, others won't.

I think this would be simple enough and actually more in line with how much responsibility and "power" we now give to account creators in terms of deciding about whether code can be upgraded, not minting duplicate NFTs or not issuing assets with and without callbacks from the same faucet.

---

Related to the discussion yesterday about whether native assets could just be standards: I think it's an okay compromise to make the compose functionality (merge/split) of the "native" assets part of the kernel so we avoid the extra work of fully generalizing assets now. However, the question of how to compose assets (+ callback flag) is really the only one that we need to answer in the tx kernel. Anything else to do with fungible and non-fungible assets could be moved out of the kernel.

A good chunk of asset-related code in the kernel has to do with native asset validation. Not a ton in terms of lines of code, but distributed to a couple of places (faucet::{mint, burn}, asset_vault::{add_asset, remove_asset}, output_note::add_asset) so it increases mental load.

The main thing I'm wondering is whether the kernel still needs to know about the structure of native assets, or only about how to compose them. For this, I'm assuming that fees are a solved problem (#2551) and the kernel does not construct the fee asset itself. Then, I believe the knowledge about the structure is only necessary for validation and the delta.

Validation
The asset preservation rules globally ensure that all assets are preserved within a given tx, so I think the only place where the validation adds real assurances is during minting and burning, where the preservation rules are basically bypassed.

For custom assets, users will already have to make sure they create valid assets themselves, so if we can give them that responsibility, we can also give them the responsibility of validating native assets. Keeping native asset validation in the kernel will in the mid-term create a weird situation where native assets are validated at lower levels (the kernel) and custom assets must be validated at higher levels. I think this will make writing uniform APIs for assets harder because one always has to keep in mind what category of asset is validated where, and this mental load applies to both the standards level and at the kernel level.

The alternative is to move this validation to miden::standards. In favor of this would be that we can establish a pattern that custom assets can mirror, e.g. for fungible assets we'd have the miden::standards::fungible_asset module, that could wrap:

• mint: validates and calls miden::protocol::faucet::mint
• burn: validates and calls miden::protocol::faucet::burn
• get: calls miden::protocol::active_account::get_asset and validates
• get_amount: calls miden::protocol::active_account::get_asset and validates and extracts amount
• add_to_account: validates and calls miden::protocol::native_account::add_asset
• remove_from_account: validates and calls miden::protocol::native_account::remove_asset
• add_to_note: validates and calls miden::protocol::output_note::add_asset

Extension by composition. Custom assets can follow the exact same pattern which is great for uniformity and therefore DevX. Of course what I proposed is not a requirement to do this, we can do this anyway. But if we do this anyway, then the kernel would really not need to do the validation.

As an example how these could be used, consider this code in BasicFungibleFaucet:

protocol/crates/miden-standards/asm/standards/faucets/mod.masm

Lines 145 to 150 in 469c811

	# mint the asset; this is needed to satisfy asset preservation logic.
	# this ensures that the asset's faucet ID matches the native account's ID.
	# this is ensured because create_fungible_asset creates the asset with the native account's ID
	exec.faucet::mint
	dropw
	# => [ASSET_KEY, ASSET_VALUE, note_idx, note_idx]

Since it works with fungible assets, it could be updated to use miden::standards::fungible_asst::mint instead of the plain mint procedure, which achieves the same level of validation.

What speaks against this approach is that malicious faucets can issue fungible assets that are invalid and bring them "into circulation". However, this problem will exist in one way or another once we have custom assets, and so the real benefit of ruling this particular case out for native assets is not clear to me.

Delta

The other part that depends tightly on the structure is the delta. We track dedicated deltas for fungible and non-fungible assets in the kernel. This means the delta is not set up for custom assets and needs a future extension. If we were to generalize assets in the delta right away, we would both simplify it and set it up for handling Word-sized custom assets in the future.

Key Strucutre

For the vault key, I would propose something like this:

/// The unique identifier of an [`Asset`] in the [`AssetVault`](crate::asset::AssetVault).
///
/// Its [`Word`] layout is:
/// ```text
/// [
///   asset_id_suffix (64 bits),
///   asset_id_prefix (64 bits),
///   [faucet_id_suffix (56 bits) | asset_structure (5 bits) | asset_compose (2 bits) | callbacks_flag (1 bit)],
///   faucet_id_prefix (64 bits)
/// ]
/// ```
pub struct AssetVaultKey {
    /// The asset ID of the vault key.
    asset_id: AssetId,

    /// The ID of the faucet that issued the asset.
    faucet_id: AccountId,

    /// Determines how assets are laid out or what structure they have.
    structure: AssetStructure,

    /// Determines the way assets are composed
    compose: AssetCompose,

    /// Determines whether callbacks are enabled.
    callbacks: AssetCallbacks,
}

pub enum AssetStructure {
    // The strucutre of assets whose value directly encodes the asset itself.
    // Initially, all assets are of this variant.
    Small,
    // Later we can add Large/Complex variants.
}

// Describes how fungible assets are composed - merged and splitted.
pub enum AssetCompose {
    // Used for assets that cannot be composed, e.g. non-fungible assets.
    None,

    // Tx kernel composes assets by using fungible_asset::{merge, split}, i.e. a "basic" way to
    // compose fungible assets.
    BasicFungible,

    // Tx kernel calls out to the faucet for composing assets.
    // Enabled in the future.
    Custom,
}

/// Whether callbacks are enabled for assets.
#[repr(u8)]
pub enum AssetCallbacks {
    #[default]
    Disabled = 0,

    Enabled = 1,
}

Notes:

• Here users can combine structure + compose procedure + callback flag. The only disallowed combination would be a future custom structure + the built-in fungible compose procedure.
• This would also allow describing custom non-fungible assets without callbacks which would not require any extra FPI call.
• We could also allocate 1-2 more bits to AssetCompose if we want to have the ability to build more "standard compose" procedures into the kernel (to avoid FPI calls for common cases). Just an idea though, not sure if this is a good one or even really possible to find general enough compose procedures.

The other impact on Rust APIs would be that the Asset definition becomes general: An AssetVaultKey + a value (Word). The AccountDelta would be simplified to match the in-kernel implementation.

Summary

To summarize: We can reduce complexity in the kernel and increase uniformity for future asset handling by moving the validation code that is specific to native assets out of the kernel and only make the kernel aware of the built-in compose functionality. This keeps the compose specialness contained to one corner of the code base instead of distributed throughout.

The benefits I think are:

• Allow writing asset APIs that make "native" assets similar to custom ones (good for uniformity). The only difference between native and custom ones would be that native ones use a built-in compose procedure, but that's one detail mostly hidden away and only relevant when creating an asset, not when using it.
• Simplifies the kernel by no longer having to validate fungible and non-fungible assets but being able to treat assets very uniformly, except for when composing assets (the one unavoidable thing).
• Simplifies the kernel by no longer having specialized delta implementations but a generalized one (this generalized one is the one described at the bottom of this post: Expand asset layout from one to two words #2328 (comment)).
• In terms of workload:
  • Ripping out the native validation from various asset kernel APIs should be relatively little work (and keep the generic validation code paths, e.g. asset key contains valid account ID). Move the fungible_asset and non_fungible_asset modules from the kernel to standards and provide some thin wrappers as proposed above. This is really just dogfooding to think about how custom assets would do validation, which can only be a benefit.
  • Refacotoring the delta is a bit more work, but still doable. It's a simplification of the existing code, not a complication.

At this point, we don't need AccountType, don't need any kind of encoding for fungible or non-fungible assets in the asset key and don't need Asset::{Fungible, NonFungible} or FungibleAssetDelta or NonFungibleAssetDelta.

Ideally, we would do this even before callbacks to avoid having to set up a different structure in that PR, though I think that would delay 0.14. The next best point in time would be right after callbacks. I would even prefer this over implementing #2176, to set us up for a much simpler kernel that we can iterate on in the future.

[bobbinth]

Overall, I like the direction of this - though, I'd probably still do callbacks now, and leave the rest for v0.15. This would give us a bit more time to think it through.

I do like the way you define the AssetVaultKey - though, for now I wouldn't introduce the AssetStructure enum (we can always do it later). So, the struct could look like so:

pub struct AssetVaultKey {
    faucet_id: AccountId,
    asset_id: AssetId,
    compose: AssetCompose,
    callbacks: AssetCallbacks,
}

pub enum AssetCompose {
    None,
    Fungible,
    Custom,
}

pub enum AssetCallbacks {
    Disabled = 0,
    Enabled = 1,
}

This does limit the flexibility a bit (i.e., we can't define multiple categories/versions of callbacks), but I think that's probably fine and in the future, we could probably handle such custom requirements via custom assets.

So, for now, we'd take up 3 bits in the 8-bit section: 1 for callbacks flag + 2 for the compose methodology flag.

[bobbinth]

Agreed, though what is the reason we need AccountType at all then? Since we no longer have protocol-reserved slots, the only distinction between the simple and faucet account type would be that one can issue assets and the other can't. But similar to account code mutability/immutability, this could also be decided at the account component level, i.e. accounts that have faucet components will be able to issue assets, others won't.

I think this would be simple enough and actually more in line with how much responsibility and "power" we now give to account creators in terms of deciding about whether code can be upgraded, not minting duplicate NFTs or not issuing assets with and without callbacks from the same faucet.

Generally agree that we could remove AccountType - but I'd probably do this in v0.15 because it will have significant amount of cascading changes and I'd rather do something like this early in a milestone rather than at the last moment (reducing the number of variants will also have cascading changes, but shouldn't be nearly as many).