zkVM Zisk Integration

[jimmychu0807]

@ArtiomTr @povi
Zisk integration is completed in this PR.
sha2 and bls12_381 is pointing to my own repository, so you may want to review and merge these two PRs:

• sha2: feat: zkvm-zisk integration (sha2) universal-precompiles#5
• bls12_381: feat: completed zisk bls12_381 patches universal-precompiles#6

and then update Cargo.toml pointing back to grandinetech repo.

[ArtiomTr]

putting here this comment just as a reminder to change it once precompiles are merged

[ArtiomTr]

don't use double encoding here -- you're currently serializing state&block&cache into SSZ, and then serialize SSZ bytes into bincode. Instead, you can pass those 3 directly as bytes into zkvm, the same as with other backends.

[jimmychu0807]

From what I found, ziskemu only takes input in a file format, not bytes directly.

• ref 1: Zisk docs
• ref 2: ere also writes the input bytes to a temp file

So the issue becomes, I either not using bincode and add byte separators and then concatenate the 5 inputs myself, or using bincode and let the lib handles this for me.

For ref, input is serialized here:
https://github.com/grandinetech/grandine/pull/475/changes#r2625605370

[ArtiomTr]

From what I found, ziskemu only takes input in a file format, not bytes directly.

Yeah, but you still can write bytes directly into file, so in that sense, it accepts bytes as input.

So the issue becomes, I either not using bincode and add byte separators and then concatenate the 5 inputs myself, or using bincode and let the lib handles this for me.

I just see it as with every other backend - you just read directly from byte array. As far as I remember, when we tried using bincode instead of ssz, we saw pretty large performance decrease. But you can try both approaches and compare, which one performs better:

• Using bincode (your current implementation).
• Doing as every other vm - reading 4 usize's from byte stream, then reading 4 slices with appropriate lengths, something like so:

  let mut buffer = [0u8; size_of::<usize>()];
  input.read_exact(&mut buffer)?;
  let state_ssz_len = usize::from_be_bytes(&v);
  
  // same for block_ssz_len, cache_ssz_len and phase_bytes_len
  
  let mut state_ssz = vec![0u8; state_ssz_len];
  input.read_exact(&mut state_ssz)?;
  
  // same for block_ssz, cache_ssz and phase_bytes

[jimmychu0807]

updated this section of code to plainly write bytes to a file and updated the guest code to read bytes from the file.

This is indeed faster than using bincode as it is simpler.

[jimmychu0807]

I want to keep this line here. Because I have never able to execute to this point actually (prove mode). So keeping this line let future dev know they could either read from the output dir (I'm not sure what's the output file name), or collect the result from the screen output.

[ArtiomTr]

I believe you can just comment out whole guest program, keep empty main, and then run prove mode locally, so that you can find where file is located.

It is better to check proving mode before merging, because after preparing all zkvms, we want to conduct some kind of report. So we will likely hand out code to zisk team and ask them to run proving on their rig.

[jimmychu0807]

Great suggestion. I am able to create the proof file. I realize Zisk proof mode doesn't output the result, so I have to run self.execute() first to get the result and the proof output.

Have also updated Proof::verify() and Proof::save() methods accordingly with the output path.

[jimmychu0807]

@ArtiomTr I have handled your comments above to the best of my knowledge. I see two remaining issues to discuss:

• piping command output to screen and capturing to variable at the same time: zkVM Zisk Integration #475 (comment)
• whether to use bincode to handle passing input to ziskVM: zkVM Zisk Integration #475 (comment)

[ArtiomTr]

Looks good, few changes needed. Also, looks like pipeline currently fails? This needs to be fixed before merging

[ArtiomTr]

From what I found, ziskemu only takes input in a file format, not bytes directly.

Yeah, but you still can write bytes directly into file, so in that sense, it accepts bytes as input.

So the issue becomes, I either not using bincode and add byte separators and then concatenate the 5 inputs myself, or using bincode and let the lib handles this for me.

I just see it as with every other backend - you just read directly from byte array. As far as I remember, when we tried using bincode instead of ssz, we saw pretty large performance decrease. But you can try both approaches and compare, which one performs better:

• Using bincode (your current implementation).
• Doing as every other vm - reading 4 usize's from byte stream, then reading 4 slices with appropriate lengths, something like so:

  let mut buffer = [0u8; size_of::<usize>()];
  input.read_exact(&mut buffer)?;
  let state_ssz_len = usize::from_be_bytes(&v);
  
  // same for block_ssz_len, cache_ssz_len and phase_bytes_len
  
  let mut state_ssz = vec![0u8; state_ssz_len];
  input.read_exact(&mut state_ssz)?;
  
  // same for block_ssz, cache_ssz and phase_bytes

[ArtiomTr]

I believe you can just comment out whole guest program, keep empty main, and then run prove mode locally, so that you can find where file is located.

It is better to check proving mode before merging, because after preparing all zkvms, we want to conduct some kind of report. So we will likely hand out code to zisk team and ask them to run proving on their rig.

[ArtiomTr]

The code looks good, however, when I try to prove locally, I get this error:

Error: Failed to copy file from "/home/necolt/workspace/grandine/zkvm/guest/zisk/artifacts/vadcop_final_proof.compressed.bin" to "/home/necolt/workspace/grandine/zkvm/host/proof.bin"

Caused by:
    No such file or directory (os error 2)

And indeed, I don't have such file:

/zkvm/guest/zisk/artifacts/vadcop_final_proof.compressed.bin

Contents of whole artifacts directory:

input.bin
result.json
vadcop_final_proof.bin

Maybe zisk won't generate .compressed proof by default?

[ArtiomTr]

You can make this error a bit nicer, by providing a bit more context:

Suggested change

	let _ = fs::copy(&proof_path, path.as_ref())
	.map_err(|e| anyhow!("Copy file failed. stderr:\n{}", e.to_string()))?;
	fs::copy(&proof_path, path.as_ref()).context(format!(
	"Failed to copy file from {proof_path:?} to {:?}",
	path.as_ref()
	))?;

Mentioning this, because I've got this error:

Error: Copy file failed. stderr:
No such file or directory (os error 2)

As you can see, it's not very helpful :). With suggested change, it looks like:

Error: Failed to copy file from "/home/necolt/workspace/grandine/zkvm/guest/zisk/artifacts/vadcop_final_proof.compressed.bin" to "/home/necolt/workspace/grandine/zkvm/host/proof.bin"

Caused by:
    No such file or directory (os error 2)

Which is much nicer in my opinion

[ArtiomTr]

to make path a bit nicer, you can do:

Suggested change

	Path::new(env!("CARGO_MANIFEST_DIR")).join("../guest/zisk")
	Path::new(env!("CARGO_MANIFEST_DIR")).parent().expect("project cannot be at root directory").join("guest/zisk")

this way paths will look better when trying to print them:

instead of:

/<project-path>/grandine/zkvm/host/../guest/zisk

you'll see:

/<project-path>/grandine/zkvm/guest/zisk

[jimmychu0807]

@ArtiomTr
Just realized Zisk no longer output compressed proof by default, and there is no CLI argument to specify that. We can do that with the SDK by converting a raw proof into compressed proof. Since I am using CLI to drive all executions here, let's drop this compressed proof and just deal with the ordinary proof output.

I have updated the rest of the PRs based on the comment and squashed all commits into one.

[ArtiomTr]

Looks good 👍

[povi]

Please don't include cryptography (or other) dependencies from a personal repo, thanks.

[jimmychu0807]

@povi Nice catch! I wasn't aware of that.

Now I have rebuilt the guest code of pico, sp1, and ziren and have the corresponding Cargo.lock updated.