Original author @thunderbiscuit
Over the past 2 weeks I've been working on a big upgrade of the Devkit Wallet, with a focus on bringing the CBF client up to what I expect for a production client. This includes putting some of the discussed ideas over dev calls in practice to see how they shake out in code, including things like waking up the phone to sync under certain conditions, for example at night and/or under wifi network access, and providing users with appropriate UI interaction and feedback to make use of the node effectively and without big surprises.
As I'm working on this a few things come to mind, and so you'll see a few new issues opened from me this week.
One Node Many Clients = No Can Do
One thing we've established quite early is that the idea of "1 node serving multiple clients" doesn't really make sense, at least for the common case where (a) you're wallet is on mainnet, and (b) you don't have 12GB of free storage you might never need again laying around. This is because the size of the filters is too great to be kept in memory or on disk in general, and therefore they must be purged after being used to check a wallet's scripts. This means wallets must perform their own download of the filters every time they are created and initially "synced", and cannot reuse a node that would keep all filters in persistence just in case another wallet needs them.
The Common Case of 1 App, Many Wallets
However, a probably common situation for mobile applications is one where the app supports many wallets (see for example Blockstream Green, Cove, Blue Wallet, etc.). Take for example an app that would hold 4 different wallets. In this case, and given our mentioned above idea of a nightly or weekly "background sync", the application finds itself performing the odd dance of waking up and starting a Kyoto node for all 4 wallets, look for peers and become one (all under the same IP address, a few seconds apart? that's another story and I don't know how peers would react to this), and download all filters for these 4 wallets. It only gets worse as more wallets are added by the user. This is basically the situation I found myself in this weekend, and how I started thinking about the cache idea described below.
The Cache As a Filter Source
This gave me an idea I'd like to riff on. What if there was a secondary valid source of filters for the node (call it KyotoCache for now) which held the previous X (configurable) filters on disk? You could provide the cache as a filter source to your Kyoto node, and could clear the cache at any point. Users would then leverage this new type like so:
- You start up the cache (a sort of modified node), download the latest X filters, and write them to a db. The cache has settings for how big you can allow it to grow, and what to do once you reach that threshold.
- When a standard Kyoto node is about to download filters from peers for new blocks it hasn't yet seen, users can set the cache as potential source for filters. The node can go and query this cache first if the cache's filters extend its best chain (the cache in this workflow is always trusted, although we could relax this assumption of course by cross-checking filter headers with peers, just not sure if that has any value if the cache is local).
- If the cache extends the node's chain, the node gets the filters from the cache. No extra network calls are made. If the cache is too shallow (the node needs 5000 blocks of filters but the cache is set to keep only 200—roughly 1ish day), the node simply proceeds as usual and queries the filters from peers.
- At any point, the cache can be cleared. In my 4-wallets example, the app would clear the
KyotoCache after the 4th wallet has used the cache, and on a weekly basis the cache would never hold more than 25ish MB of data.
This sort of construct would be quite narrow-focused and wouldn't apply to many use cases, but would make the use case of "1 app, many wallets" potentially cleaner if they tend to all sync together, as they probably would on mobile.
This also potentially makes the app a better peer by not requiring download of all filters for the week as many times as the app has wallets, although the individual wallets will potentially still have to connect to peers and download blocks individually. Still, the number of times this has to be done is also reduced since not all wallets will need blocks for the given sync window, ensuring we create less new peers than we currently do.
Original author @thunderbiscuit
Over the past 2 weeks I've been working on a big upgrade of the Devkit Wallet, with a focus on bringing the CBF client up to what I expect for a production client. This includes putting some of the discussed ideas over dev calls in practice to see how they shake out in code, including things like waking up the phone to sync under certain conditions, for example at night and/or under wifi network access, and providing users with appropriate UI interaction and feedback to make use of the node effectively and without big surprises.
As I'm working on this a few things come to mind, and so you'll see a few new issues opened from me this week.
One Node Many Clients = No Can Do
One thing we've established quite early is that the idea of "1 node serving multiple clients" doesn't really make sense, at least for the common case where (a) you're wallet is on mainnet, and (b) you don't have 12GB of free storage you might never need again laying around. This is because the size of the filters is too great to be kept in memory or on disk in general, and therefore they must be purged after being used to check a wallet's scripts. This means wallets must perform their own download of the filters every time they are created and initially "synced", and cannot reuse a node that would keep all filters in persistence just in case another wallet needs them.
The Common Case of 1 App, Many Wallets
However, a probably common situation for mobile applications is one where the app supports many wallets (see for example Blockstream Green, Cove, Blue Wallet, etc.). Take for example an app that would hold 4 different wallets. In this case, and given our mentioned above idea of a nightly or weekly "background sync", the application finds itself performing the odd dance of waking up and starting a Kyoto node for all 4 wallets, look for peers and become one (all under the same IP address, a few seconds apart? that's another story and I don't know how peers would react to this), and download all filters for these 4 wallets. It only gets worse as more wallets are added by the user. This is basically the situation I found myself in this weekend, and how I started thinking about the cache idea described below.
The Cache As a Filter Source
This gave me an idea I'd like to riff on. What if there was a secondary valid source of filters for the node (call it
KyotoCachefor now) which held the previous X (configurable) filters on disk? You could provide the cache as a filter source to your Kyoto node, and could clear the cache at any point. Users would then leverage this new type like so:KyotoCacheafter the 4th wallet has used the cache, and on a weekly basis the cache would never hold more than 25ish MB of data.This sort of construct would be quite narrow-focused and wouldn't apply to many use cases, but would make the use case of "1 app, many wallets" potentially cleaner if they tend to all sync together, as they probably would on mobile.
This also potentially makes the app a better peer by not requiring download of all filters for the week as many times as the app has wallets, although the individual wallets will potentially still have to connect to peers and download blocks individually. Still, the number of times this has to be done is also reduced since not all wallets will need blocks for the given sync window, ensuring we create less new peers than we currently do.