Conversation
|
Hi @ybc-alkaid. Thanks for your contribution. The riscv-arch-test repository recently migrated to ACT4 (version 4 of the test suite) and is no longer adding new extensions to the old 3.0 framework. Please review the guidelines in the Developer Guide for instructions on writing tests and coverpoints for the act4 framework. Unfortunately, there is likely to be some significant work involved in porting these tests. Please reach out with any questions or documentation suggestions and let us know if you need help porting the tests. |
|
@jordancarlin, @jamesbeyond, @UmerShahidengr, can you please do a preliminary review and let me know if this appears to be workable, i.e. yes work remains like porting to ACT4, but the coverage and quality of the tests appear to be reasonably complete. |
|
This PR opens a bunch of great questions about what new extensions need in ACT. There are 21 tests in this PR. At a high level, they look like good tests exercising the basic functions of SPMP. There are a lot of things missing to fit into the ACT4 framework:
Also, the number of tests is much lower in this PR than in ACT4 PMP. It tests each feature individually, but lacks cross-products. It has RV64, but not yet RV32. On a cursory inspection, it seems it does not stress edge cases such as access size, misaligned accesses and accesses partially contained within a region. It does not stress interaction with other extensions, such as compressed, atomics, cache management ops, hypervisor, and vector. For example, the spec contains normative rules about when hypervisor is subject to SPMP checks, and that SPMP is mutually exclusive with virtual memory, that do not seem to be part of these tests. This brings up important questions about how extensive tests should be for new extensions, and how to run them while simulator support is also evolving. |
@davidharrishmc, I agree. Because ACT4 is now the main branch, we should being requesting PRs to meet our needs.
@UmerShahidengr, can you share the user documentation with @ybc-alkaid so that he know how to adjust his tests for the framework?
@ybc-alkaid, these tests are important and we want to do the work properly. Thank you for your patience here. |
|
@jjscheel @ybc-alkaid Most of this is documented in the Developer Guide, but the documentation is still fairly new and we definitely need to add more to it. We will continue to expand it as we get questions and suggestions from contributors, but it should be a sufficient starting place. Feel free to open issues or come to the ACT SIG meeting every other Thursday (including this week) if you have questions about any of it. |
@davidharrishmc can you help us list the requirements to we can embed this in the spec dev process? @jordancarlin maybe you can also help here? |
|
@jjscheel @rpsene @jordancarlin @davidharrishmc |
|
+Jordan, James, Umer, Allen, Jamie, Moiz
To support a new extension with the same level of testing and test documentation that CSC has been targeting for ratified extensions, here is the start of a requirements list. It’s pretty extensive and will obviously need review by others and feedback from early adopters, and cleanup to be easily used. There will be a significant transition time before all this is ready. It might be worth stabilizing RVA23 certificate tests (including parameters, UDB, CI, etc) before asking new extensions to even attempt doing all this.
* Merge the extension chapter into riscv-isa-manual/src. This presents a chicken-and-egg problem about merging and ratification. Right now we don’t have tooling to handle normative rule tagging for chapters that are not in the riscv-isa-manual repo, but if rules and tests are a requirement to ratify an extensions, we need a new way to handle them before they are merged.
* Tag normative rules in the riscv-isa-manual/src/<Extension>.adoc chapter per
https://github.com/riscv/riscv-isa-manual/blob/main/tagging_normative_rules.adoc
* Add a YAML file in riscv-isa-manual/normative_rule_defs/<Extension>.yaml mapping the tags to normative rules
* Compile the ISA manual. Visually inspect that malformed tags didn’t introduce garbles. Review the hyperlinked normative rules. <James Ball could flush out this checklist with details and best practices.>
* Implement support for the extension into at least Sail. Preferred to also support Spike, QEMU, Whisper.
* Define the extension and its parameters in UDB. This is still being flushed out by the Parameters Working Group.
* The remaining artifacts are generally in riscv-arch-test:
* For “unprivileged” tests whose testplans are driven by CSV files indicating which coverpoints apply to which instructions:
- Add a testsplans/<Extension>.csv defining the instructions and applicable coverpoints
- Modify generators to support new coverpoints <Jordan could flush this out>
- Radically new extensions such as matrix might need entirely new test generator scripts
* For “privileged” tests described by Google Sheets spreadsheets
- Create human-readable test plan spreadsheet in the form of
https://docs.google.com/spreadsheets/d/1VuBt-H9VVWh8k-USAJEplrgEJETjQ-iNjD4Iq_ZOnN0/edit?gid=1474055416#gid=1474055416
- These spreadsheets need to migrate into an RVI directory and be put into a form with standardized columns.
- Write coverpoints by hand in coverpoints/priv/<Extension>.svh
- Write a test generator in generators/testgen/src/testgen/priv/extensions (see docs/DeveloperGuide.md)
- Coverpoints and tests may depend on parameters
* Extensions with significantly new types of architectural state may need to extend RVVI and the Sail RVVI tracer to support the coverpoints.
- RVVI is a standard driven by Synopsys. A process needs to be in place to propose and ratify extensions.
* Test the tests
- Review of coverpoints/<priv, unpriv>/Extension.svh
- Review tests/<priv, unpriv>/Extension
- Run “make coverage” and ensure 100% coverpoint coverage
- Check that tests pass on all supported simulators (<Jordan flush this out>), and preferably on at least one RTL proof-of-concept core.
- For extensions with parameters, we need to define a way to test an interesting range of parameters
* Update CTP documentation:
- Add coverpoints/norm/<Extension>.yaml listing the names of each normative rule and one or more coverpoints that show the rule has been exercised.
- Edit docs/ctp/src/*.adoc similar to other extensions, with at least a summary of new coverpoints and include files to the autogenerated testplan .adoc and normative rule .adoc. Describe parameters if applicable.
- Update docs/ctp/src/profiles.adoc listing the new extension and hyperlinking to the docs. When a profile is added, also update this file indicating which extensions are needed in the profile.
* Continuous Integration
- Need a process to bring tests into CI, including on configurations with parameter edge cases
- Need a way to allow CI waivers while infrastructure such as simulators are in development, and ensure the waivers are removed when the infrastructure is ready to run the tests.
Questions:
- How does RVI train members to efficiently do all of these steps? A lot of domain knowledge is required in specialized areas including enhancing multiple simulators, normative rules, UDB, RVVI, and ACT.
- Does RVI need a staff member with the expertise in all of these areas to work with the domain expert driving the new extension?
- Access to licensed RTL simulator with functional coverage for coverpoint development and CI.
- Managing versions of all these repositories (simulators, UDB, ISA manual, ACT, etc) so a compatible collection of branches exist to fully test the new extension before it is ratified and before all pieces are merged.
- Reviewing and merging all of these branches in a timely way after the extension is ratified. This has been a problem for ACT, and will be a bigger problem when there are dependencies across repos with different owners and stringent expectations of code maturity.
- Tracking all of the artifacts and dependencies, ensuring the new extension runs in CI on multiple simulators in a timely way.
- Developing all of this imposes a high cost on adding extensions, and the implications deserve TSC or board review.
David
… On Mar 17, 2026, at 11:45 PM, Rafael Sene ***@***.***> wrote:
rpsene
left a comment
(riscv/riscv-arch-test#1069)
<#1069 (comment)>
This PR opens a bunch of great questions about what new extensions need in ACT.
There are 21 tests in this PR. At a high level, they look like good tests exercising the basic functions of SPMP.
There are a lot of things missing to fit into the ACT4 framework:
Human-readable spreadsheet testplan
Normative rules
SystemVerilog coverpoints
Mapping of normative rules onto coverpoints
Parameters
Porting tests to ACT4 framework in self-checking form
Support for Sail, Spike, QEMU
Chicken and egg problem here developing the simulator to produce the expected results
Also, the number of tests is much lower in this PR than in ACT4 PMP. It tests each feature individually, but lacks cross-products. It has RV64, but not yet RV32. On a cursory inspection, it seems it does not stress edge cases such as access size, misaligned accesses and accesses partially contained within a region. It does not stress interaction with other extensions, such as compressed, atomics, cache management ops, hypervisor, and vector. For example, the spec contains normative rules about when hypervisor is subject to SPMP checks, and that SPMP is mutually exclusive with virtual memory, that do not seem to be part of these tests.
This brings up important questions about how extensive tests should be for new extensions, and how to run them while simulator support is also evolving.
@davidharrishmc <https://github.com/davidharrishmc> can you help us list the requirements to we can embed this in the spec dev process? @jordancarlin <https://github.com/jordancarlin> maybe you can also help here?
—
Reply to this email directly, view it on GitHub <#1069 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AR4AA3ZN75NGSCOA52UEYOT4RFQITAVCNFSM6AAAAACWNY2VWCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DANZVGUYTQNBYHE>.
You are receiving this because you were mentioned.
|
|
+Garry
Regards,
Jamie Grimm
From: David Harris ***@***.***> On Behalf Of David Harris
Sent: Tuesday, March 17, 2026 12:32 PM
To: riscv/riscv-arch-test ***@***.***>
Cc: riscv/riscv-arch-test ***@***.***>; Mention ***@***.***>; Jordan Carlin ***@***.***>; James Ball ***@***.***>; Umer Shahid ***@***.***>; Jamie Grimm ***@***.***>; Moiz Hussain ***@***.***>; Allen Baum ***@***.***>
Subject: Re: [riscv/riscv-arch-test] Add tests for Sspmp extension (PR #1069)
+Jordan, James, Umer, Allen, Jamie, Moiz
To support a new extension with the same level of testing and test documentation that CSC has been targeting for ratified extensions, here is the start of a requirements list. It’s pretty extensive and will obviously need review by others and feedback from early adopters, and cleanup to be easily used. There will be a significant transition time before all this is ready. It might be worth stabilizing RVA23 certificate tests (including parameters, UDB, CI, etc) before asking new extensions to even attempt doing all this.
* Merge the extension chapter into riscv-isa-manual/src. This presents a chicken-and-egg problem about merging and ratification. Right now we don’t have tooling to handle normative rule tagging for chapters that are not in the riscv-isa-manual repo, but if rules and tests are a requirement to ratify an extensions, we need a new way to handle them before they are merged.
* Tag normative rules in the riscv-isa-manual/src/<Extension>.adoc chapter per
https://github.com/riscv/riscv-isa-manual/blob/main/tagging_normative_rules.adoc
* Add a YAML file in riscv-isa-manual/normative_rule_defs/<Extension>.yaml mapping the tags to normative rules
* Compile the ISA manual. Visually inspect that malformed tags didn’t introduce garbles. Review the hyperlinked normative rules. <James Ball could flush out this checklist with details and best practices.>
* Implement support for the extension into at least Sail. Preferred to also support Spike, QEMU, Whisper.
* Define the extension and its parameters in UDB. This is still being flushed out by the Parameters Working Group.
* The remaining artifacts are generally in riscv-arch-test:
* For “unprivileged” tests whose testplans are driven by CSV files indicating which coverpoints apply to which instructions:
- Add a testsplans/<Extension>.csv defining the instructions and applicable coverpoints
- Modify generators to support new coverpoints <Jordan could flush this out>
- Radically new extensions such as matrix might need entirely new test generator scripts
* For “privileged” tests described by Google Sheets spreadsheets
- Create human-readable test plan spreadsheet in the form of
https://docs.google.com/spreadsheets/d/1VuBt-H9VVWh8k-USAJEplrgEJETjQ-iNjD4Iq_ZOnN0/edit?gid=1474055416#gid=1474055416
- These spreadsheets need to migrate into an RVI directory and be put into a form with standardized columns.
- Write coverpoints by hand in coverpoints/priv/<Extension>.svh
- Write a test generator in generators/testgen/src/testgen/priv/extensions (see docs/DeveloperGuide.md)
- Coverpoints and tests may depend on parameters
* Extensions with significantly new types of architectural state may need to extend RVVI and the Sail RVVI tracer to support the coverpoints.
- RVVI is a standard driven by Synopsys. A process needs to be in place to propose and ratify extensions.
* Test the tests
- Review of coverpoints/<priv, unpriv>/Extension.svh
- Review tests/<priv, unpriv>/Extension
- Run “make coverage” and ensure 100% coverpoint coverage
- Check that tests pass on all supported simulators (<Jordan flush this out>), and preferably on at least one RTL proof-of-concept core.
- For extensions with parameters, we need to define a way to test an interesting range of parameters
* Update CTP documentation:
- Add coverpoints/norm/<Extension>.yaml listing the names of each normative rule and one or more coverpoints that show the rule has been exercised.
- Edit docs/ctp/src/*.adoc similar to other extensions, with at least a summary of new coverpoints and include files to the autogenerated testplan .adoc and normative rule .adoc. Describe parameters if applicable.
- Update docs/ctp/src/profiles.adoc listing the new extension and hyperlinking to the docs. When a profile is added, also update this file indicating which extensions are needed in the profile.
* Continuous Integration
- Need a process to bring tests into CI, including on configurations with parameter edge cases
- Need a way to allow CI waivers while infrastructure such as simulators are in development, and ensure the waivers are removed when the infrastructure is ready to run the tests.
Questions:
- How does RVI train members to efficiently do all of these steps? A lot of domain knowledge is required in specialized areas including enhancing multiple simulators, normative rules, UDB, RVVI, and ACT.
- Does RVI need a staff member with the expertise in all of these areas to work with the domain expert driving the new extension?
- Access to licensed RTL simulator with functional coverage for coverpoint development and CI.
- Managing versions of all these repositories (simulators, UDB, ISA manual, ACT, etc) so a compatible collection of branches exist to fully test the new extension before it is ratified and before all pieces are merged.
- Reviewing and merging all of these branches in a timely way after the extension is ratified. This has been a problem for ACT, and will be a bigger problem when there are dependencies across repos with different owners and stringent expectations of code maturity.
- Tracking all of the artifacts and dependencies, ensuring the new extension runs in CI on multiple simulators in a timely way.
- Developing all of this imposes a high cost on adding extensions, and the implications deserve TSC or board review.
David
On Mar 17, 2026, at 11:45 PM, Rafael Sene ***@***.******@***.***>> wrote:
[https://avatars.githubusercontent.com/u/20800562?s=20&v=4]rpsene left a comment (riscv/riscv-arch-test#1069)<#1069 (comment)>
This PR opens a bunch of great questions about what new extensions need in ACT.
There are 21 tests in this PR. At a high level, they look like good tests exercising the basic functions of SPMP.
There are a lot of things missing to fit into the ACT4 framework:
* Human-readable spreadsheet testplan
* Normative rules
* SystemVerilog coverpoints
* Mapping of normative rules onto coverpoints
* Parameters
* Porting tests to ACT4 framework in self-checking form
* Support for Sail, Spike, QEMU
* Chicken and egg problem here developing the simulator to produce the expected results
Also, the number of tests is much lower in this PR than in ACT4 PMP. It tests each feature individually, but lacks cross-products. It has RV64, but not yet RV32. On a cursory inspection, it seems it does not stress edge cases such as access size, misaligned accesses and accesses partially contained within a region. It does not stress interaction with other extensions, such as compressed, atomics, cache management ops, hypervisor, and vector. For example, the spec contains normative rules about when hypervisor is subject to SPMP checks, and that SPMP is mutually exclusive with virtual memory, that do not seem to be part of these tests.
This brings up important questions about how extensive tests should be for new extensions, and how to run them while simulator support is also evolving.
@davidharrishmc<https://github.com/davidharrishmc> can you help us list the requirements to we can embed this in the spec dev process? @jordancarlin<https://github.com/jordancarlin> maybe you can also help here?
—
Reply to this email directly, view it on GitHub<#1069 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AR4AA3ZN75NGSCOA52UEYOT4RFQITAVCNFSM6AAAAACWNY2VWCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DANZVGUYTQNBYHE>.
You are receiving this because you were mentioned.Message ID: ***@***.******@***.***>>
|
|
So, @ybc-alkaid, your request is very valid: we need to decompose the full list that @davidharrishmc has posted into those items needed BEFORE any extension is becomes a part of a test plan. From that list, we can split that into 2 lists: those which are needed for Freeze and those which need to completed afterward. Historically, ACT test required a coverpoints and tests in a PR. The whole test framework has changed and it been extended to support "validation" of the coverage by using Verilog coverpoints when used in the context of our certification processes. The referenced document from @jordancarlin (Developer Guide) provides a great overview of the current framework. What we need to do is review it to ensure we've made clear what needs to be done for tests not yet in a Certificate. I'd like to ask @UmerShahidengr to work with @jordancarlin and @davidharrishmc and @jamesbeyond to work on this and then provide specific guidance which we can work into the document as needed. Does this sound like a reasonable plan to all? |
|
Most (if not all) of those steps are required whether an extension is currently part of a certificate or not. For example, the Sm tests have gone through all of these steps even though they are not currently part of a certificate. We definitely need to review which parts are needed for freeze, but I think that eventually all of these pieces will be needed for all extensions independent of certification. |
|
The whole discussion sounds like that process and framework on compatibility test/framework isnt in very solid state. Appreciate the desire and efforts to get this better over time. However in order to keep operations ongoing, we need an RVI-accepted intermediate approach, which still allows us to drive standardization/specification work further. @jordancarlin: Can we use one of the ACT-SIG session to align on the proceedings to be used, till all methodology and framework is readily defined?
|
Let's discuss this at the next ACT SIG meeting on 4/2. I think we only have one other thing on the agenda at the moment. |
Ok |
7 participants
This PR introduces basic tests for the RISC-V S-level Physical Memory Protection (SPMP) extension.
Here is a pre-built PDF of SPMP:
https://github.com/riscv/riscv-spmp/blob/main/rv-spmp-spec.pdf
Here is a pre-built PDF of riscv-privileged manual with SPMP:
https://github.com/riscv/riscv-spmp/releases/download/v0.8.18/riscv-privileged-with-spmp.pdf
Here is the pointer to the SPMP specification:
riscv/riscv-isa-manual#2573
Here is the Spike implementation supporting the tests:
riscv-software-src/riscv-isa-sim#2244
Here are the test logs:
https://drive.google.com/drive/folders/1MT65PQ7_RgEeZ8O90XkNR9AzX2AzLgIe?usp=drive_link