VM crashes silently when performing execute over `mainnet without epoch transition`

[Dyslex7c]

When I perform execute operation on test case pectra-devnet-6 with/without epoch transition, it is working as intended and returning the state root after state transition. However, it throws an error on test case mainnet without epoch transition

Running test "mainnet without epoch transition"
stdout: loading block and state...

thread 'main' panicked at zkvm/host/src/main.rs:145:59:
called `Result::unwrap()` on an `Err` value: []
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Due to the following line where it returns an empty object to the output

let (output, report) = client.execute(STATE_TRANSITION_ELF, stdin).run()?;
ZKMPublicValues { buffer: Buffer { data: [], ptr: 0 } }

I printed out the report as well but couldn't make out anything from it

report: ExecutionReport { opcode_counts: {ADD: 625899249, SUB: 23512521, MULT: 1, MULTU: 673159, MUL: 1346318, DIV: 0, DIVU: 8, SLL: 302908115, SRL: 2181, SRA: 0, ROR: 8, SLT: 0, SLTU: 182457880, AND: 84595197, OR: 172554799, XOR: 738814, NOR: 27, CLZ: 39, CLO: 0, BEQ: 260731798, BGEZ: 0, BGTZ: 0, BLEZ: 0, BLTZ: 673184, BNE: 174140366, Jump: 57504016, Jumpi: 90365337, JumpDirect: 0, LB: 1, LBU: 52733427, LH: 0, LHU: 3898, LW: 750256967, LWL: 58623824, LWR: 58623824, LL: 24, SB: 38087070, SH: 2112, SW: 719180158, SWL: 39402289, SWR: 39402289, SC: 24, SYSCALL: 7, MEQ: 6612196, MNE: 12944187, TEQ: 8, SEXT: 0, WSBH: 8, EXT: 0, MADDU: 0, MSUBU: 0, INS: 8, MOD: 0, MODU: 0, UNIMPL: 0}, syscall_counts: {SYSHINTLEN: 3, SYSHINTREAD: 3, SYSVERIFY: 0, HALT: 0, WRITE: 1, ENTER_UNCONSTRAINED: 0, EXIT_UNCONSTRAINED: 0, SHA_EXTEND: 0, SHA_COMPRESS: 0, ED_ADD: 0, ED_DECOMPRESS: 0, KECCAK_SPONGE: 0, SECP256K1_ADD: 0, SECP256K1_DOUBLE: 0, SECP256K1_DECOMPRESS: 0, BN254_ADD: 0, BN254_DOUBLE: 0, COMMIT: 0, COMMIT_DEFERRED_PROOFS: 0, VERIFY_ZKM_PROOF: 0, BLS12381_DECOMPRESS: 0, UINT256_MUL: 0, U256XU2048_MUL: 0, BLS12381_ADD: 0, BLS12381_DOUBLE: 0, BLS12381_FP_ADD: 0, BLS12381_FP_SUB: 0, BLS12381_FP_MUL: 0, BLS12381_FP2_ADD: 0, BLS12381_FP2_SUB: 0, BLS12381_FP2_MUL: 0, BN254_FP_ADD: 0, BN254_FP_SUB: 0, BN254_FP_MUL: 0, BN254_FP2_ADD: 0, BN254_FP2_SUB: 0, BN254_FP2_MUL: 0, SECP256R1_ADD: 0, SECP256R1_DOUBLE: 0, SECP256R1_DECOMPRESS: 0, POSEIDON2_PERMUTE: 0, UNIMPLEMENTED: 0}, cycle_tracker: {}, touched_memory_addresses: 0 }

Most likely, it crashes while carrying out the following deserializations, indicating that maybe it doesn't support the large input size provided

let block = SignedBeaconBlock::<P>::from_ssz(config, &block_ssz)?;
let state = BeaconState::<P>::from_ssz(config, &state_ssz)?;
let cache = PubkeyCache::from_ssz(config, &cache_ssz)?;

Is there a way to enable debug output to see what exactly is the problem here? Because currently it is hard to figure out what went wrong without proper error message

[felicityin]

How can I reproduce this issue?

[Dyslex7c]

Here is the entire codebase, I have added you to this.
https://github.com/Dyslex7c/grandine-zk/tree/ziren

Clone the ziren branch of the repo, then modify the guest code in zkvm/guest/ziren/src/main.rs by replacing

let config = Config::pectra_devnet_6();

with

let config = Config::mainnet();

Then navigate to the directory zkvm/guest/ziren and follow the instructions given in the README.md file over there. Then from the root directory, run

cargo run -p zkvm_host --features ziren --release -- --test "mainnet without epoch transition" execute

[felicityin]

@weilzkm Please help take a look at this issue—this line of code executes successfully on the host but fails on the guest.

However, the guest program exited without reporting any errors.

The state_ssz occupies 264.25 MB - could this be causing excessive memory usage?

[Dyslex7c]

The state_ssz occupies 264.25 MB - could this be causing excessive memory usage?

Could be but I think it is less likely. Does ziren use bump-allocator? Maybe it fails because the allocated memory limit is lower. In that case, we might require an alternative allocator

[weilzkm]

Yeah，the default allocator is bump-allocator, but the memory overflowed when I changed it to embeded-allocator.

Could be but I think it is less likely. Does ziren use bump-allocator? Maybe it fails because the allocated memory limit is lower. In that case, we might require an alternative allocator

[Dyslex7c]

Any information on the current memory limits? We encountered similar issues with SP1 which has a limit of around 2GB and is able to read mainnet input fully, but not execute STF.

[eigmax]

@Dyslex7c do you mean SP1 can read 2GB for a single stdin::read? I think you need to enable the embedded feature?

We also support embedded feature, but can only read 1GB each time, example: https://github.com/ProjectZKM/Ziren/blob/main/examples/large-sum/host/src/main.rs#L14

[Dyslex7c]

do you mean SP1 can read 2GB for a single stdin::read?

Not sure but I think they meant 2GB is the highest amount of memory they can accommodate and a single input likely takes a few hundred megabytes. You can check their issue here succinctlabs/sp1#2334

I think you need to enable the embedded feature?

Fine but I guess @weilzkm has tested that and got memory overflow as he mentioned in a previous comment

[weilzkm]

Yeah. I have tried the embeded mode. It seems the total memory is not enough to run this test. Our max memory is 0x7F000000. It's a little bigger than sp1, I think. Can you give a smaller input to test?@Dyslex7c

[Dyslex7c]

@weilzkm yes we have tested with pectra-devnet-6 with/without epoch transition which I believe has a smaller input and it works. This issue occurs only when using mainnet

[Dyslex7c]

@weilzkm @eigmax Hey can we get access to a GPU prover for Ziren because proving locally takes a lot of time (~12 hours) before I have to stop it eventually

[eigmax]

Do you still want to use our GPU prover?

[Dyslex7c]

yes it'd be nice if we could use the GPU prover

[eigmax]

Do you have GPU card? if you have, you can run it by yourself. Or we can provide you an access key.

[Dyslex7c]

GPU card nope. I'm on apple silicon mac M3