tests: remove flash validation error failure condition

[PatrickRobbIOL]

Description

Previously, some tests within the flash validation testsuite were set to fail when the error prefix was detected. This is now removed per instruction from the Ocre community developers. Now, the flash validation testsuite includes only one testcase which just checks for the expected hello world string and fails if it is not detected.

Type of change

Please delete options that are not relevant.

• CI system update
• Test Coverage update

How Has This Been Tested?

This will be tested when the PR is opened, given that it is a change to how the GHA workflows are set up.

Checklist:

• I have performed a self-review of my own code

[kr-t]

Hi @PatrickRobbIOL , shouldn't this now be green: https://github.com/project-ocre/ocre-runtime/actions/runs/18349248688
Can you check please, why it isn't successful?

[PatrickRobbIOL]

Hi @PatrickRobbIOL , shouldn't this now be green: https://github.com/project-ocre/ocre-runtime/actions/runs/18349248688 Can you check please, why it isn't successful?

Hi Krystian, yes tests.yml should be passing now. I see 2 issues:

1. All 3 tests are running. This is not right, as 2 of the 3 tests have been removed from the flashValidation group, and also removed from the config.json for the flashValidation group. I am trying to figure out the reason. It may be that the checkout stage is checking out to the main branch instead of the feature branch which is the source for the PR. I will send a new commit to test this.

2. Hello World is not in the "stdout" when a break is sent for the b_u5 board. I'm not sure why this is happening - when we first introduced the tests the hello world container was running on startup. This is what the logs for the tests.yml run show:

---

Beginning test group Flash Validation
Test the Ocre runtime after the flash is complete

Entering Setup...

Script: Flash Validation Setup

Setup is complete

Entering Testing...

Test Suite: Runtime Validation Tests

Test Case: Check Runtime Hello World

**** READING RESPONSE FROM BREAK ****

I: OSPI flash config is OPI / DTR
I: Read SFDP from octoFlash
I: Read SFDP from octoFlash
I: W5500 Initialized
I: chip id 0x6b
I: pressure: int on gpio@42021800.02
*** Booting Zephyr OS build 8d0d392f8cc7 ***
Waiting for network to be ready...

ocre:~$
**** CLOSING CONNECTION ****

TEST FAILED

---

I wonder, could this be the result of networking being introduced? After a certain timeout, will the "waiting for network to be ready..." timeout and the Hello World container will be run? Or perhaps the build procedure in build.yml was modified in the past months to no longer include the Hello World container by default. Anyhow, I will try to figure out why the Hello World container is not running anymore.

[kr-t]

Hi @PatrickRobbIOL , the networking is implemented in a way that it shall not be a breaking change. If we don't have internet, we just print it and go on with the flow. Check this log:

This is an U585 without the internet connection - as you can see, after being Unable to connect to network, we still run the container.

[PatrickRobbIOL]

Hi @PatrickRobbIOL , the networking is implemented in a way that it shall not be a breaking change. If we don't have internet, we just print it and go on with the flow. Check this log:

Thank you. My PR is now passing the hello world test. The reason it was not working is because it was reading the stdout when the board was halfway through initializing. So, it was capturing something from the first half (the networking setup) but it was too early to capture the "Hello World from Ocre." I think the reason why this used to work with the old timeout is that less was happening during the ocre init, and now that more is happening, a longer sleep is needed.

I also had to move the tests from tests.yml to build.yml because of restrictions related to workflow_run.

[kr-t]

I can see that hello world passes. Shall we remove the test.yml then? Because as of now we still have them failing, which we probably don't want as our default state. Or would you suggest changing something else? Thanks!

[PatrickRobbIOL]

I can see that hello world passes. Shall we remove the test.yml then? Because as of now we still have them failing, which we probably don't want as our default state. Or would you suggest changing something else? Thanks!

Hi. Right, tests.yml is being removed in this PR. This is desired from my perspective as I want to move the invocation of the tests to build.yml.

The tests.yml test is still running (until this PR is merged) because it runs based on the main branch, not the PR source branch.

[kr-t]

Hi @PatrickRobbIOL I'd merge it, but you have to have verified signature on github, otherwise it isn't letting me.
https://docs.github.com/en/authentication/managing-commit-signature-verification/displaying-verification-statuses-for-all-of-your-commits

[PatrickRobbIOL]

Thanks. I actually wasn't aware of key signing git commits. Cool!

So, this PR should be unblocked now. If we can merge this, early next week Matt and I can send in some new testcases. He showed me his testcase for the modbus server today and it looks good.

[kr-t]

Hi @PatrickRobbIOL , the flashing test isn't working anymore. I think there is nothing wrong with the test itself, but the way we flash the board. If you check the earlier test, it was fine (on right) - but now somehow it isn't loading the correct image (left). Check the sizes of the containers - on right it's the correct 3850, on left some giant size of something else. I see two possible problems: either you tested something else, and the debugger aren't clearing correctly the flash - or you have some other zephyr.bin which it flashes instead if the image it should.

P.S. I have downloaded and tested the artifact from this PR - it works fine on my board. So something is messed up either during the automatic download/storage or the flash itself.
I'd suggest rewriting the flash command to:
STM32_Programmer_CLI -c port=swd -e all -w zephyr.bin 0x08000000 -v -rst
-e all would erase the flash before writing it, this might solve the issue if there is some garbage left.

We need to rely on this test and on the board, otherwise no pull request will be able to pass it if it won't work deterministically.

[PatrickRobbIOL]

Hi @PatrickRobbIOL , the flashing test isn't working anymore. I think there is nothing wrong with the test itself, but the way we flash the board. If you check the earlier test, it was fine (on right) - but now somehow it isn't loading the correct image (left). Check the sizes of the containers - on right it's the correct 3850, on left some giant size of something else. I see two possible problems: either you tested something else, and the debugger aren't clearing correctly the flash - or you have some other zephyr.bin which it flashes instead if the image it should.

P.S. I have downloaded and tested the artifact from this PR - it works fine on my board. So something is messed up either during the automatic download/storage or the flash itself. I'd suggest rewriting the flash command to: STM32_Programmer_CLI -c port=swd -e all -w zephyr.bin 0x08000000 -v -rst -e all would erase the flash before writing it, this might solve the issue if there is some garbage left.

We need to rely on this test and on the board, otherwise no pull request will be able to pass it if it won't work deterministically.

Thanks, yes I looked this morning and I saw the new logs, and that instead of starting the Hello World container, it says the container fails to start. Matt will look at it today.

Matt, first things first let me update the PR so that it uses Krystian's recommended command.

[kr-t]

Hi Patrick, it seems that command might have worked - new commit passes with no issues.
Let me retrigger the actions to check if they would pass once more - if yes, we can merge this and focus on new stuff.

[kr-t]

It seems that it is working, even though I had to retrigger it again. Let's deliver it, but I propose that in the future we review our testing and separate them into multiple files/jobs.