Skip to content

accdb: wip concurrent cache #8616

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
ripatel-fd wants to merge 1 commit into
base: main
Choose a base branch
from
+2,559 −8,462
Draft

accdb: wip concurrent cache #8616

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 19 additions & 12 deletions book/api/metrics-generated.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -610,6 +610,12 @@
| <span class="metrics-name">execle_&#8203;transaction_&#8203;landed</span><br/>{transaction_&#8203;landed="<span class="metrics-enum">landed_&#8203;failed</span>"} | counter | Whether a transaction landed in the block or not (Transaction landed, but failed to execute) |
| <span class="metrics-name">execle_&#8203;transaction_&#8203;landed</span><br/>{transaction_&#8203;landed="<span class="metrics-enum">unlanded</span>"} | counter | Whether a transaction landed in the block or not (Transaction did not land) |
| <span class="metrics-name">execle_&#8203;compute_&#8203;units_&#8203;total</span> | counter | Estimated number of compute units executed since tile start |
| <span class="metrics-name">execle_&#8203;accdb_&#8203;lookup_&#8203;funk</span> | counter | Number of account lookups resolved from funk (in-memory fork store) |
| <span class="metrics-name">execle_&#8203;accdb_&#8203;lookup_&#8203;specrd</span> | counter | Number of account lookups resolved from speculative read (vinyl cache) |
| <span class="metrics-name">execle_&#8203;accdb_&#8203;lookup_&#8203;accdb</span> | counter | Number of account lookups sent to accdb tile (vinyl rq/cq) |
| <span class="metrics-name">execle_&#8203;accdb_&#8203;dt_&#8203;funk</span> | counter | Cumulative time spent in funk (in-memory) account lookups |
| <span class="metrics-name">execle_&#8203;accdb_&#8203;dt_&#8203;specrd</span> | counter | Cumulative time spent in speculative read (vinyl cache) account lookups |
| <span class="metrics-name">execle_&#8203;accdb_&#8203;dt_&#8203;vinyl</span> | counter | Cumulative time spent waiting for vinyl rq/cq account lookups |

</div>

Expand Down Expand Up @@ -925,15 +931,12 @@
| <span class="metrics-name">replay_&#8203;progcache_&#8203;gc_&#8203;root</span> | counter | Number of program cache entries garbage collected while rooting |
| <span class="metrics-name">replay_&#8203;accdb_&#8203;created</span> | counter | Number of account database records created |
| <span class="metrics-name">replay_&#8203;accdb_&#8203;reverted</span> | counter | Number of account database records reverted |
| <span class="metrics-name">replay_&#8203;accdb_&#8203;rooted</span> | counter | Number of account database entries rooted |
| <span class="metrics-name">replay_&#8203;accdb_&#8203;rooted_&#8203;bytes</span> | counter | Number of bytes in account database entries rooted (including overhead) |
| <span class="metrics-name">replay_&#8203;accdb_&#8203;gc_&#8203;root</span> | counter | Number of account database entries garbage collected |
| <span class="metrics-name">replay_&#8203;accdb_&#8203;reclaimed</span> | counter | Number of account database entries reclaimed (deletion rooted) |
| <span class="metrics-name">replay_&#8203;root_&#8203;slot_&#8203;duration_&#8203;seconds</span> | histogram | Time in seconds spent updating the rooted account store (one sample per block) |
| <span class="metrics-name">replay_&#8203;root_&#8203;account_&#8203;duration_&#8203;seconds</span> | histogram | Time in seconds spent updating the rooted account store (one sample per block, normalized by account count) |
| <span class="metrics-name">replay_&#8203;root_&#8203;elapsed_&#8203;seconds</span><br/>{root_&#8203;phase="<span class="metrics-enum">db</span>"} | counter | Total time in seconds spent rooting accounts (Waiting on database server) |
| <span class="metrics-name">replay_&#8203;root_&#8203;elapsed_&#8203;seconds</span><br/>{root_&#8203;phase="<span class="metrics-enum">copy</span>"} | counter | Total time in seconds spent rooting accounts (Copying account data) |
| <span class="metrics-name">replay_&#8203;root_&#8203;elapsed_&#8203;seconds</span><br/>{root_&#8203;phase="<span class="metrics-enum">gc</span>"} | counter | Total time in seconds spent rooting accounts (Garbage collecting old account data) |
| <span class="metrics-name">replay_&#8203;accdb_&#8203;lookup_&#8203;funk</span> | counter | Number of account lookups resolved from funk (in-memory fork store) |
| <span class="metrics-name">replay_&#8203;accdb_&#8203;lookup_&#8203;specrd</span> | counter | Number of account lookups resolved from speculative read (vinyl cache) |
| <span class="metrics-name">replay_&#8203;accdb_&#8203;lookup_&#8203;accdb</span> | counter | Number of account lookups sent to accdb tile (vinyl rq/cq) |
| <span class="metrics-name">replay_&#8203;accdb_&#8203;dt_&#8203;funk</span> | counter | Cumulative time spent in funk (in-memory) account lookups |
| <span class="metrics-name">replay_&#8203;accdb_&#8203;dt_&#8203;specrd</span> | counter | Cumulative time spent in speculative read (vinyl cache) account lookups |
| <span class="metrics-name">replay_&#8203;accdb_&#8203;dt_&#8203;vinyl</span> | counter | Cumulative time spent waiting for vinyl rq/cq account lookups |

</div>

Expand All @@ -951,6 +954,12 @@
| <span class="metrics-name">execrp_&#8203;progcache_&#8203;dup_&#8203;inserts</span> | counter | Number of time two tiles raced to insert the same cache entry |
| <span class="metrics-name">execrp_&#8203;progcache_&#8203;invalidations</span> | counter | Number of program cache invalidations |
| <span class="metrics-name">execrp_&#8203;accdb_&#8203;created</span> | counter | Number of account database records created |
| <span class="metrics-name">execrp_&#8203;accdb_&#8203;lookup_&#8203;funk</span> | counter | Number of account lookups resolved from funk (in-memory fork store) |
| <span class="metrics-name">execrp_&#8203;accdb_&#8203;lookup_&#8203;specrd</span> | counter | Number of account lookups resolved from speculative read (vinyl cache) |
| <span class="metrics-name">execrp_&#8203;accdb_&#8203;lookup_&#8203;accdb</span> | counter | Number of account lookups sent to accdb tile (vinyl rq/cq) |
| <span class="metrics-name">execrp_&#8203;accdb_&#8203;dt_&#8203;funk</span> | counter | Cumulative time spent in funk (in-memory) account lookups |
| <span class="metrics-name">execrp_&#8203;accdb_&#8203;dt_&#8203;specrd</span> | counter | Cumulative time spent in speculative read (vinyl cache) account lookups |
| <span class="metrics-name">execrp_&#8203;accdb_&#8203;dt_&#8203;vinyl</span> | counter | Cumulative time spent waiting for vinyl rq/cq account lookups |
| <span class="metrics-name">execrp_&#8203;txn_&#8203;regime</span><br/>{txn_&#8203;regime="<span class="metrics-enum">setup</span>"} | counter | Mutually exclusive and exhaustive duration of time spent in transaction execution regimes (Transaction setup) |
| <span class="metrics-name">execrp_&#8203;txn_&#8203;regime</span><br/>{txn_&#8203;regime="<span class="metrics-enum">exec</span>"} | counter | Mutually exclusive and exhaustive duration of time spent in transaction execution regimes (Transaction execution (includes VM setup/execution)) |
| <span class="metrics-name">execrp_&#8203;txn_&#8203;regime</span><br/>{txn_&#8203;regime="<span class="metrics-enum">commit</span>"} | counter | Mutually exclusive and exhaustive duration of time spent in transaction execution regimes (Transaction result commit) |
Expand Down Expand Up @@ -994,9 +1003,7 @@
| <span class="metrics-name">accdb_&#8203;bstream_&#8203;seq</span><br/>{bstream_&#8203;seq="<span class="metrics-enum">present</span>"} | gauge | Current bstream sequence number (Blocks between present and future are being written (write only)) |
| <span class="metrics-name">accdb_&#8203;bstream_&#8203;seq</span><br/>{bstream_&#8203;seq="<span class="metrics-enum">future</span>"} | gauge | Current bstream sequence number (Blocks between future and ancient have not been written (no read, no write)) |
| <span class="metrics-name">accdb_&#8203;request_&#8203;batches</span> | counter | Number of request batches processed |
| <span class="metrics-name">accdb_&#8203;requests</span><br/>{vinyl_&#8203;request="<span class="metrics-enum">acquire</span>"} | counter | Number of requests processed (Acquire record) |
| <span class="metrics-name">accdb_&#8203;requests</span><br/>{vinyl_&#8203;request="<span class="metrics-enum">release</span>"} | counter | Number of requests processed (Release record) |
| <span class="metrics-name">accdb_&#8203;requests</span><br/>{vinyl_&#8203;request="<span class="metrics-enum">erase</span>"} | counter | Number of requests processed (Erase record) |
| <span class="metrics-name">accdb_&#8203;requests</span> | counter | Number of requests processed |
| <span class="metrics-name">accdb_&#8203;blocks</span><br/>{vinyl_&#8203;blocks="<span class="metrics-enum">pair</span>"} | counter | Number of blocks written to bstream (Record) |
| <span class="metrics-name">accdb_&#8203;blocks</span><br/>{vinyl_&#8203;blocks="<span class="metrics-enum">dead</span>"} | counter | Number of blocks written to bstream (Record deletion) |
| <span class="metrics-name">accdb_&#8203;blocks</span><br/>{vinyl_&#8203;blocks="<span class="metrics-enum">part</span>"} | counter | Number of blocks written to bstream (Partition/divider) |
Expand Down
23 changes: 21 additions & 2 deletions src/app/firedancer-dev/commands/backtest.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
#include "../../../disco/topo/fd_topob_vinyl.h"
#include "../../../util/pod/fd_pod_format.h"
#include "../../../discof/genesis/fd_genesi_tile.h"
#include "../../../funk/fd_funk_base.h"
#include "../../../discof/replay/fd_replay_tile.h"
#include "../../../discof/restore/fd_snapin_tile_private.h"
#include "../../../discof/restore/fd_snaplv_tile_private.h"
Expand Down Expand Up @@ -90,6 +91,11 @@ backtest_topo( config_t * config ) {
ulong funk_locks_obj_id; FD_TEST( (funk_locks_obj_id = fd_pod_query_ulong( topo->props, "funk_locks", ULONG_MAX ) )!=ULONG_MAX );
fd_topob_tile_uses( topo, replay_tile, &topo->objs[ funk_obj_id ], FD_SHMEM_JOIN_MODE_READ_WRITE );
fd_topob_tile_uses( topo, replay_tile, &topo->objs[ funk_locks_obj_id ], FD_SHMEM_JOIN_MODE_READ_WRITE );
if( vinyl_enabled ) {
fd_topo_tile_t * accdb_tile = &topo->tiles[ fd_topo_find_tile( topo, "accdb", 0UL ) ];
fd_topob_tile_uses( topo, accdb_tile, &topo->objs[ funk_obj_id ], FD_SHMEM_JOIN_MODE_READ_WRITE );
fd_topob_tile_uses( topo, accdb_tile, &topo->objs[ funk_locks_obj_id ], FD_SHMEM_JOIN_MODE_READ_WRITE );
}

fd_topob_wksp( topo, "progcache" );
setup_topo_progcache( topo, "progcache",
Expand Down Expand Up @@ -349,8 +355,10 @@ backtest_topo( config_t * config ) {
setup_topo_accdb_meta( topo, &config->firedancer );
ulong vinyl_map_obj_id = fd_pod_query_ulong( topo->props, "accdb.meta_map", ULONG_MAX ); FD_TEST( vinyl_map_obj_id !=ULONG_MAX );
ulong vinyl_pool_obj_id = fd_pod_query_ulong( topo->props, "accdb.meta_pool", ULONG_MAX ); FD_TEST( vinyl_pool_obj_id!=ULONG_MAX );
ulong vinyl_line_obj_id = fd_pod_query_ulong( topo->props, "accdb.line", ULONG_MAX ); FD_TEST( vinyl_line_obj_id!=ULONG_MAX );
fd_topo_obj_t * accdb_map_obj = &topo->objs[ vinyl_map_obj_id ];
fd_topo_obj_t * accdb_pool_obj = &topo->objs[ vinyl_pool_obj_id ];
fd_topo_obj_t * accdb_line_obj = &topo->objs[ vinyl_line_obj_id ];

fd_topo_obj_t * accdb_data = setup_topo_accdb_cache( topo, &config->firedancer );

Expand All @@ -359,13 +367,20 @@ backtest_topo( config_t * config ) {
fd_topob_tile_uses( topo, accdb_tile, accdb_data, FD_SHMEM_JOIN_MODE_READ_WRITE );
fd_topob_tile_uses( topo, accdb_tile, accdb_map_obj, FD_SHMEM_JOIN_MODE_READ_WRITE );
fd_topob_tile_uses( topo, accdb_tile, accdb_pool_obj, FD_SHMEM_JOIN_MODE_READ_WRITE );
fd_topob_tile_uses( topo, accdb_tile, accdb_line_obj, FD_SHMEM_JOIN_MODE_READ_WRITE );

fd_topob_tile_uses( topo, replay_tile, accdb_data, FD_SHMEM_JOIN_MODE_READ_WRITE );
fd_topob_tile_uses( topo, replay_tile, accdb_data, FD_SHMEM_JOIN_MODE_READ_WRITE );
fd_topob_tile_uses( topo, replay_tile, accdb_line_obj, FD_SHMEM_JOIN_MODE_READ_WRITE );
for( ulong i=0UL; i<execrp_tile_cnt; i++ ) {
fd_topob_tile_uses( topo, &topo->tiles[ fd_topo_find_tile( topo, "execrp", i ) ], accdb_data, FD_SHMEM_JOIN_MODE_READ_WRITE );
fd_topo_tile_t * t = &topo->tiles[ fd_topo_find_tile( topo, "execrp", i ) ];
fd_topob_tile_uses( topo, t, accdb_data, FD_SHMEM_JOIN_MODE_READ_WRITE );
fd_topob_tile_uses( topo, t, accdb_line_obj, FD_SHMEM_JOIN_MODE_READ_WRITE );
}

fd_topob_wksp( topo, "accdb_replay" );

fd_topob_wksp( topo, "replay_accdb" );
fd_topob_link( topo, "replay_accdb", "replay_accdb", 128UL, sizeof(fd_funk_txn_xid_t), 1UL );
}

/**********************************************************************/
Expand Down Expand Up @@ -415,6 +430,10 @@ backtest_topo( config_t * config ) {
fd_topob_wksp( topo, "replay_execrp" );
fd_topob_link( topo, "replay_execrp", "replay_execrp", 16384UL, 2240UL, 1UL );
fd_topob_tile_out( topo, "replay", 0UL, "replay_execrp", 0UL );
if( vinyl_enabled ) {
fd_topob_tile_out( topo, "replay", 0UL, "replay_accdb", 0UL );
fd_topob_tile_in( topo, "accdb", 0UL, "metric_in", "replay_accdb", 0UL, FD_TOPOB_RELIABLE, FD_TOPOB_POLLED );
}
for( ulong i=0UL; i<execrp_tile_cnt; i++ ) {
fd_topob_tile_in( topo, "execrp", i, "metric_in", "replay_execrp", 0UL, FD_TOPOB_RELIABLE, FD_TOPOB_POLLED );
}
Expand Down
2 changes: 2 additions & 0 deletions src/app/firedancer-dev/main.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ extern fd_topo_obj_callbacks_t fd_obj_cb_rnonce_ss;

extern fd_topo_obj_callbacks_t fd_obj_cb_vinyl_meta;
extern fd_topo_obj_callbacks_t fd_obj_cb_vinyl_meta_ele;
extern fd_topo_obj_callbacks_t fd_obj_cb_vinyl_line;
extern fd_topo_obj_callbacks_t fd_obj_cb_vinyl_data;
extern fd_topo_obj_callbacks_t fd_obj_cb_vinyl_req_pool;
extern fd_topo_obj_callbacks_t fd_obj_cb_vinyl_rq;
Expand All @@ -54,6 +55,7 @@ fd_topo_obj_callbacks_t * CALLBACKS[] = {
&fd_obj_cb_funk_locks,
&fd_obj_cb_vinyl_meta,
&fd_obj_cb_vinyl_meta_ele,
&fd_obj_cb_vinyl_line,
&fd_obj_cb_vinyl_data,
&fd_obj_cb_acc_pool,
&fd_obj_cb_vinyl_req_pool,
Expand Down
28 changes: 28 additions & 0 deletions src/app/firedancer/callbacks_vinyl.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,34 @@ fd_topo_obj_callbacks_t fd_obj_cb_vinyl_meta_ele = {
.new = vinyl_meta_ele_new,
};

/* vinyl_line: cache line array shared between accdb tile and clients */

static ulong
vinyl_line_align( fd_topo_t const * topo,
fd_topo_obj_t const * obj ) {
(void)topo; (void)obj;
return alignof(fd_vinyl_line_t);
}

static ulong
vinyl_line_footprint( fd_topo_t const * topo,
fd_topo_obj_t const * obj ) {
return sizeof(fd_vinyl_line_t) * VAL("line_cnt");
}

static void
vinyl_line_new( fd_topo_t const * topo,
fd_topo_obj_t const * obj ) {
fd_memset( fd_topo_obj_laddr( topo, obj->id ), 0, vinyl_line_footprint( topo, obj ) );
}

fd_topo_obj_callbacks_t fd_obj_cb_vinyl_line = {
.name = "vinyl_line",
.footprint = vinyl_line_footprint,
.align = vinyl_line_align,
.new = vinyl_line_new,
};

/* vinyl_data: memory arena for data cache entries */

static ulong
Expand Down
4 changes: 2 additions & 2 deletions src/app/firedancer/config/default.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -543,7 +543,7 @@ telemetry = true
# how much memory is reserved for such account changes.
# If the amount of inflight account changes exceeds this limit, the
# validator will crash.
max_unrooted_account_size_gib = 16
max_unrooted_account_size_gib = 64

# Keep frequently accessed accounts in memory to improve
# performance.
Expand Down Expand Up @@ -579,7 +579,7 @@ telemetry = true
# The expected mean size of recently accessed accounts.
#
# This heuristic is used to pack accounts into caches optimally.
mean_account_footprint = 256
mean_account_footprint = 64

# io_uring specific options
#
Expand Down
4 changes: 2 additions & 2 deletions src/app/firedancer/config/mainnet.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,8 @@
[consensus]
expected_genesis_hash = "5eykt4UsFv8P8NJdTREpY1vzqKqZKvdpKuc147dw2N9d"
[accounts]
file_size_gib = 600
max_accounts = 1_100_000_000
file_size_gib = 700
max_accounts = 1_300_000_000
[snapshots]
[snapshots.sources]
servers = [ "http://solana-mainnet-rpc.jumpisolated.com:8899" ]
2 changes: 2 additions & 0 deletions src/app/firedancer/main.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ extern fd_topo_obj_callbacks_t fd_obj_cb_rnonce_ss;

extern fd_topo_obj_callbacks_t fd_obj_cb_vinyl_meta;
extern fd_topo_obj_callbacks_t fd_obj_cb_vinyl_meta_ele;
extern fd_topo_obj_callbacks_t fd_obj_cb_vinyl_line;
extern fd_topo_obj_callbacks_t fd_obj_cb_vinyl_data;
extern fd_topo_obj_callbacks_t fd_obj_cb_vinyl_req_pool;
extern fd_topo_obj_callbacks_t fd_obj_cb_vinyl_rq;
Expand All @@ -53,6 +54,7 @@ fd_topo_obj_callbacks_t * CALLBACKS[] = {
&fd_obj_cb_acc_pool,
&fd_obj_cb_vinyl_meta,
&fd_obj_cb_vinyl_meta_ele,
&fd_obj_cb_vinyl_line,
&fd_obj_cb_vinyl_data,
&fd_obj_cb_acc_pool,
&fd_obj_cb_vinyl_req_pool,
Expand Down
Loading
Loading