Skip to content

support generate checkpoint to flash file and memory file, and support restore from these files#714

Open
xyyy1420 wants to merge 40 commits intomasterfrom
protobuf_and_restore_device
Open

support generate checkpoint to flash file and memory file, and support restore from these files#714
xyyy1420 wants to merge 40 commits intomasterfrom
protobuf_and_restore_device

Conversation

@xyyy1420
Copy link
Contributor

@xyyy1420 xyyy1420 commented Dec 10, 2024 •
edited by shinezyy
Loading

  • Submodule
    • add protobuf
  • take cpt to flash
    • support for generating checkpoints to flash, thus avoiding the need to link payloads with libcheckpointAlpha.
    • two files will be generated, a flash image and a memory image
    • support difftest from flash
    • change the restore option to specify a flash image, since both checkpoint images and binary are essentially memory mirrors
  • difftest
    • support override ref model pmp and pmpcfg regs
    • support override ref model flash contents
    • tried to fix difftest attach but failed, because of spike has many non-register states which are hard to fix
  • CI
    • because protobuf is added, submodule is initialized by default in CI
    • add take cpt with flash and restore cpt with flash
    • migrating nutshell-spike-ref from local to ready-to-run
  • config file
    • fix the address setting in NEMU riscv64-nutshell-diff-spike_defconfig according to the flash address set in spike.
    • add two configuration files for generating checkpoints to be placed in flash
  • MMIO
    • add three flags to indicate which operations need to skip difftest ref model

@xyyy1420 xyyy1420 requested a review from shinezyy December 10, 2024 09:19
@xyyy1420
Copy link
Contributor Author

xyyy1420 commented Dec 10, 2024

load gcpt to flash need position-indenpendent code OpenXiangShan/LibCheckpointAlpha#6

@xyyy1420 xyyy1420 force-pushed the protobuf_and_restore_device branch 4 times, most recently from 32d4183 to ddf2930 Compare December 10, 2024 10:40
@xyyy1420
Copy link
Contributor Author

xyyy1420 commented Dec 16, 2024

OpenXiangShan/riscv-isa-sim#70 this pr fixes the difftest after checkpoint restore

@xyyy1420 xyyy1420 marked this pull request as draft December 19, 2024 02:25
@xyyy1420 xyyy1420 force-pushed the protobuf_and_restore_device branch from 2f10b4f to 266e786 Compare December 19, 2024 03:22
@xyyy1420
Copy link
Contributor Author

xyyy1420 commented Dec 19, 2024

OpenXiangShan/ready-to-run#78 Bump spike ref in ready-to-run

@xyyy1420
Copy link
Contributor Author

xyyy1420 commented Dec 19, 2024

OpenXiangShan/ready-to-run#79 add nutshell spike so to ready-to-run

@xyyy1420
Copy link
Contributor Author

xyyy1420 commented Dec 20, 2024

OpenXiangShan/riscv-isa-sim#71 support force override vtype

@xyyy1420
Copy link
Contributor Author

xyyy1420 commented Dec 20, 2024

OpenXiangShan/ready-to-run#80 bump spike ref in ready-to-run

@xyyy1420
Copy link
Contributor Author

xyyy1420 commented Dec 25, 2024

OpenXiangShan/riscv-isa-sim#75 support override spike flash

@xyyy1420
delete useless code
49baea5 
Signed-off-by: jiaxiaoyu <xyyy1420@gmail.com>
@xyyy1420
checkpoint: add gcpt device, cpu regs will store in device mmio space,
255a606 
add protobuf, using for encode checkpoint memlayout

Signed-off-by: jiaxiaoyu <xyyy1420@gmail.com>
@xyyy1420
checkpoint: add using_gcpt_mmio arg
ce06338 
Signed-off-by: jiaxiaoyu <xyyy1420@gmail.com>
@xyyy1420
checkpoint: add compress gcpt and pmem space
663dcd0 
Signed-off-by: jiaxiaoyu <xyyy1420@gmail.com>
@xyyy1420
is_gz/zstd_file: panic support filename
704ff3c 
Signed-off-by: jiaxiaoyu <xyyy1420@gmail.com>
@xyyy1420
finish zstd/gz checkpoint using gcpt device and restore from device
16da779 
Signed-off-by: jiaxiaoyu <xyyy1420@gmail.com>
@xyyy1420
fix ng checkpoint
9372be8 
Signed-off-by: jiaxiaoyu <xyyy1420@gmail.com>
@xyyy1420
feat(libcheckpoint): add support for libcheckpoint
492aa13 
Signed-off-by: jiaxiaoyu <xyyy1420@gmail.com>
@xyyy1420
feat(libcheckpoint): migrate to libcheckpoint memlayout
46ecdc2 
Signed-off-by: jiaxiaoyu <xyyy1420@gmail.com>
@xyyy1420
feat(checkpoint): support generate checkpoint to memory archive and f…
ba303bf 
…lash archive, and support load checkpoint from these files

Signed-off-by: jiaxiaoyu <xyyy1420@gmail.com>
@xyyy1420
feat(cpt-defconfig): support next-generation checkpoint by default
3c23325 
Signed-off-by: jiaxiaoyu <xyyy1420@gmail.com>
@xyyy1420
feat(checkpoint): support libcheckpoint memlayout need init submodule…
5d6509d 
… 'protobuf' first

Signed-off-by: jiaxiaoyu <xyyy1420@gmail.com>
@xyyy1420
fix(flash): fix bugs using flash devices
f5c5933 
Signed-off-by: jiaxiaoyu <xyyy1420@gmail.com>
@xyyy1420
set limit, just using next generation checkpoint when flash device en…
0b71c23 
…able

Signed-off-by: jiaxiaoyu <xyyy1420@gmail.com>
@xyyy1420
fix take checkpoint script
0108e0a 
Signed-off-by: jiaxiaoyu <xyyy1420@gmail.com>
@xyyy1420
fix(difftest attach): add pmp and pmp cfg copy to difftest attach
62b4f6a 
Signed-off-by: jiaxiaoyu <xyyy1420@gmail.com>
@xyyy1420
fix(config): update config for more accurate instruction counting
9cfad1a 
Signed-off-by: jiaxiaoyu <xyyy1420@gmail.com>
@xyyy1420
fix(jrelop): check difftest attch when exec jrelop
f15cc7c 
Signed-off-by: jiaxiaoyu <xyyy1420@gmail.com>
@xyyy1420
Bump submodule(ready-to-run): Bump spike ref, nemu ref in ready-to-run
01c5f7c 
Signed-off-by: jiaxiaoyu <xyyy1420@gmail.com>
@xyyy1420
fix(difftest): Enable pmp register copy only when flash exists
fc5e44a 
Signed-off-by: jiaxiaoyu <xyyy1420@gmail.com>
@xyyy1420
fix(ci): The restore parameter has changed its function and is used t…
8a5a36d 
…o specify the checkpoint file.

Signed-off-by: jiaxiaoyu <xyyy1420@gmail.com>
@xyyy1420
feat(ci): Replace the local spike-so used by nutshell with the spike-…
f04e4c2 
…so of ready-to-run

Signed-off-by: jiaxiaoyu <xyyy1420@gmail.com>
@xyyy1420
feat(difftest): add detach and attach log
a055d5c 
Signed-off-by: jiaxiaoyu <xyyy1420@gmail.com>
@xyyy1420
fix(ci): The restore parameter has changed its function and is used t…
c013291 
…o specify the checkpoint file.

Signed-off-by: jiaxiaoyu <xyyy1420@gmail.com>
@xyyy1420
fix(ci): Delete intermediates
f796a25 
Signed-off-by: jiaxiaoyu <xyyy1420@gmail.com>
@xyyy1420
Bump submodule(ready-to-run): Bump spike ref in ready-to-run
0513f74 
Signed-off-by: jiaxiaoyu <xyyy1420@gmail.com>
@xyyy1420
debug
8264153 
Signed-off-by: jiaxiaoyu <xyyy1420@gmail.com>
@xyyy1420 xyyy1420 force-pushed the protobuf_and_restore_device branch from 141efc5 to aed3ce1 Compare December 25, 2024 08:15
@xyyy1420
feat(difftest): Supports selective difftest of mmio address space
923297c 
Signed-off-by: jiaxiaoyu <xyyy1420@gmail.com>
@xyyy1420
delete useless output
1771d44 
Signed-off-by: jiaxiaoyu <xyyy1420@gmail.com>
@xyyy1420
delete debug code
ee6b1ec 
Signed-off-by: jiaxiaoyu <xyyy1420@gmail.com>
@xyyy1420
bump ready-to-run
aed3ce1 
Signed-off-by: jiaxiaoyu <xyyy1420@gmail.com>
@xyyy1420
fix compile in centos docker
dbf91da 
Signed-off-by: jiaxiaoyu <xyyy1420@gmail.com>
@xyyy1420
fix ci error
ff7fd5f 
Signed-off-by: jiaxiaoyu <xyyy1420@gmail.com>
@xyyy1420
fix ci error
27a7019 
Signed-off-by: jiaxiaoyu <xyyy1420@gmail.com>
@xyyy1420
fix mmio skip diff flag
e3ebffe 
Signed-off-by: jiaxiaoyu <xyyy1420@gmail.com>
@xyyy1420
fix mmio skip difftest ref
0230e0f 
Signed-off-by: jiaxiaoyu <xyyy1420@gmail.com>
@xyyy1420
fix mmio map
c023a91 
Signed-off-by: jiaxiaoyu <xyyy1420@gmail.com>
@xyyy1420 xyyy1420 marked this pull request as ready for review December 26, 2024 07:11
poemonsense
poemonsense reviewed Dec 26, 2024
View reviewed changes
Copy link
Member

@poemonsense poemonsense left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is difficult for me to review such a large PR. Is it possible to split it into multiple smaller PRs?

@xyyy1420
Copy link
Contributor Author

xyyy1420 commented Dec 26, 2024

It is difficult for me to review such a large PR. Is it possible to split it into multiple smaller PRs?

Okay, I'm going to split it and request a review again.

@cebarobot
Copy link
Member

cebarobot commented Dec 26, 2024

I agree with great Poemonsense. It's better to split this large PR into smaller ones.

@xyyy1420 xyyy1420 mentioned this pull request Dec 26, 2024
Merged
@xyyy1420 xyyy1420 added the do not merge please do not merge it until something label Jan 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@poemonsense poemonsense poemonsense left review comments

@shinezyy shinezyy Awaiting requested review from shinezyy

@cebarobot cebarobot Awaiting requested review from cebarobot cebarobot is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

do not merge please do not merge it until something

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@xyyy1420 @cebarobot @poemonsense