GSOC 2025 Final evalutation: refactor SHV to newly added shv-libs4c library, add image updates and come up with generated code abstraction #118
Conversation
ppisa
left a comment
ppisa
left a comment
There was a problem hiding this comment.
Please correct duplicated lines from some merge.
zdebanos
force-pushed
the
gsoc-2025-shv-refactor
branch
3 times, most recently
from
3af2886 to
0a4b0d7
Compare
| return | ||
|
|
||
| queue.put_nowait("Confirming the image on fwStable/confirm.") | ||
| await asyncio.sleep(0) |
There was a problem hiding this comment.
It happened that the queue (the cosument of messages printed to the textbox) got stuck, so i needed to yield so it completed the print.
There was a problem hiding this comment.
But I think this could be solved if I try to rewrite the .py file to be compatible with pyshv>=0.10.0. Also, I'm not very familiar with asyncio and I think there is a lot of room for improvement.
| from supsisim.port import Port, InPort, OutPort | ||
| from supsisim.connection import Connection | ||
| from supsisim.dialg import RTgenDlg, SHVDlg | ||
| from supsisim.dialg import RTgenDlg, SHVDlg, UpdimgDlg |
There was a problem hiding this comment.
Just a nitpick, but I would use UpdImgDlg instead of UpdimgDlg
There was a problem hiding this comment.
I kinda like UpdimgDlg.
There was a problem hiding this comment.
I am fine with it as long as linter doesn't complain
|
Code generation on Linux (rt target) seems to be broken now. Libs compiled with |
|
@michallenc Working on LinuxRT rn. |
zdebanos
force-pushed
the
gsoc-2025-shv-refactor
branch
2 times, most recently
from
6793f9f to
b8c2bdb
Compare
|
@michallenc The pipeline has passed. I've fixed |
|
@michallenc Also, the target you provided compiles OK, and you can also tune the model using SHV, and updates work as well ( |
|
@michallenc i've also stumbled accross using pyshv==0.10.0, i think i'll need to rewrite the whole |
zdebanos
force-pushed
the
gsoc-2025-shv-refactor
branch
from
9f82246 to
ef39d28
Compare
I am not sure where the issue is there. The old |
|
Trying to test the patch on both Linux and NuttX and having troubles to get correct results... Linux RT target now compiles and runs without errors, but it doesn't mount itself to the broker. Testing with pyshv 0.10.1 against pyshvbroker run as On NuttX target, I was unable to generate the code, seems like a link problems with SHV lib. I am not using SHV lib compiled with NuttX, but external downloaded by pysimCoder makefile. Maybe there is the issue? The board configuration is |
zdebanos
force-pushed
the
gsoc-2025-shv-refactor
branch
from
ef39d28 to
44be159
Compare
|
Fixed. The NuttX makefile had a typo in it. Also, it pulled the wrong branch (the dev one), instead of master. |
michallenc
left a comment
michallenc
left a comment
There was a problem hiding this comment.
Tested NuttX target, it successfully builds, runs and mounts to the broker.
zdebanos
force-pushed
the
gsoc-2025-shv-refactor
branch
5 times, most recently
from
91dbeb9 to
df61208
Compare
…ts SHV CI The functional CI is critical during massive SH update which does not touch Arduino Firmata at all Signed-off-by: Pavel Pisa <pisa@fel.cvut.cz>
…irectory The actual location where common part of the rules is located is CodeGen/nuttx/rules/make-rules.inc Signed-off-by: Stepan Pressl <pressl.stepan@gmail.com> Signed-off-by: Pavel Pisa <pisa@fel.cvut.cz>
… version The external libraries are cloned/fetched according to the version specified by ExtLibs/config where the specifications are stored in *-git.url and *-version.sha files. The uLAN Utilities Library (uLUt) is used as an example. Signed-off-by: Pavel Pisa <pisa@fel.cvut.cz>
zdebanos
force-pushed
the
gsoc-2025-shv-refactor
branch
from
9448e79 to
7fda133
Compare
…on sources download The external libraries are cloned/fetched according to the version specified ExtLibs/config where the specifications are stored in *-git.url and *-version.sha files. The build from the common sources is resolved for the specific target OUTPUT_DIR in the OMK build cases. Libraries and generated/exported include headers are stored into include-generated directory for given target. The template then takes includes and libraries from this location. Linux RT, Linux MZ_APO and NuttX have been ported for now. Signed-off-by: Pavel Pisa <pisa@fel.cvut.cz>
…uttX export Signed-off-by: Pavel Pisa <pisa@fel.cvut.cz>
Signed-off-by: Stepan Pressl <pressl.stepan@gmail.com> Signed-off-by: Pavel Pisa <pisa@fel.cvut.cz>
…code This commit adds many changes to pysimCoder. 1) The whole SHV infrastructure has been reworked to a unified library called shv-libs4c. Basically, this library is based on the pysimCoder SHV routines, but now it's unified, provides transport layer abstraction (now, we support TCP/IP, only), provides platform abstraction (NuttX, Linux). In future, it could support more embedded devices and operating systems. Some old files were deleted in CodeGen/Common/shv, whereas some new were added - especially those responsible for the new SHV features. 2) Image update dialog has been added. Currently, there are 2 options. The first one is straightforward: using openocd to flash a device. This is applicable only to embedded devices. The second method are the SHV File node updates. Useful especially when working with NuttX when NXBoot is utilized. The dialog continously sends chunks of a new NuttX image to a NuttX device with some external storage, used as an update paritition by NXBoot. If the flashing is succesful (after a CRC check), the device is resetted. In the end, the image must be confirmed, as that is the part of the NXBoot API. 3) Big shv/generator.py rework. This is due to the firmware NXBoot updates. New nodes are added. 4) Overall rework and proof of concept of code generation. I've introduced the concept of model's context and manager. With this I'd want to rework all sources that it is possible to have multiple models running without any global variables. The manager is an abstract interface to the model, it allows to pause or resume the model's execution. The reason we want to pause the model's execution are the previously mentioned firmware updates. If a high task control loop is running with strict deadlines, some CPU load responsible for SHV communication and interaction with the update partition may cause these deadlines to be missed. It's just better to stop the model's execution at all, do some communication, and then restart (or resume). 5) Template makefile changes, where SHV is utilized (NuttX, LinuxRT) SHV cloning. Added stadartized makefile steps (CC, AR, ...) Currently, nuttx.tmf, nuttx_timerhook.tmf, rt.tmf and rt_mz_apo.tmf generated targets (i.e. the only supporting SHV) are stable. Despite the fact that other targets should compile OK as there are no changes to the core generator, these targets need to be reworked aswell. Signed-off-by: Stepan Pressl <pressl.stepan@gmail.com>
- wrong include guard in the manager node - SHVBytes include fix - additional parameters of the fwStable and manager methods - added get method to fwStable Signed-off-by: Stepan Pressl <pressl.stepan@gmail.com>
Signed-off-by: Stepan Pressl <pressl.stepan@gmail.com>
Signed-off-by: Stepan Pressl <pressl.stepan@gmail.com>
Signed-off-by: Stepan Pressl <pressl.stepan@gmail.com>
Signed-off-by: Stepan Pressl <pressl.stepan@gmail.com>
…th it If NuttX is configured with NETUTILS_LIBSHVC then use exported includes and libraries and do not build the local copy. Signed-off-by: Pavel Pisa <pisa@fel.cvut.cz>
zdebanos
force-pushed
the
gsoc-2025-shv-refactor
branch
from
f74837b to
cc6fa77
Compare
|
Hello @robertobucher, I consider that @zdebanos massive work and contribution reached state which is worthy for broader testing. Important is to check if something out of our scope has been broken. I have tested on Linux RT, NuttX and Xilinx Zynq MZ_APO. Even all intermediate commits are tested that all these platforms and three corresponding example models build and in Linux RT case even run. NuttX on physical hardware has been tested only at some key points in the series and then at the final state. It would worth probably to reuse and test Raspberry Pi with SHV support, deices Makefile and TMF from Linux RT or MZ_APO TMF. @zdebanos works on re-introduction option to specify SHV broker address and login parameters from environment variables. But it is not critical for basic testing and can be added even latter. It would worth to organize some online meeting as well, but at least this week till Thursday is completely impossible on my side. |
If environment variables handling is not supported in NuttX, the Makefile raises an error. Signed-off-by: Stepan Pressl <pressl.stepan@gmail.com>
This main.c now allows for overriding of SHV default arguments. Signed-off-by: Stepan Pressl <pressl.stepan@gmail.com>
This main.c now allows for overrding of SHV default parameters. Signed-off-by: Stepan Pressl <pressl.stepan@gmail.com>
Signed-off-by: Stepan Pressl <pressl.stepan@gmail.com>
zdebanos
force-pushed
the
gsoc-2025-shv-refactor
branch
from
ea1323d to
5863125
Compare
|
@ppisa @michallenc Thank you for your reviews, I've added the introduction of overridable environment variables into |
|
I would accept single |
|
I also forgot to thank @ppisa for helping me rebase and divide the commits into smaller chunks. |
4 participants
Added functionalities
In the commit message.
This commit also proposes some abstraction over the generated code. My approach is now only experimental, but the code generator (and SHV generator, as well) are ready to generate code for a binary, where multiple models will be ran. Each model will have its own context (a dynamically allocated platform dependant struct) and NO global variables should be present in platform respective mains, all should be contained within the structs.
The reason I came up with the abstraction is explained in the commit message, point n.4.
Disclaimer
Now, it only works with NuttX. LinuxRT's devices compile with the newest shv-libs4c library but I need to make the codegen working.
I did not rework
nuttx_main_systemtickhook.c, as I thinknuttx_main_timerhook.cis much better, has less overhead and I have invested a lot of time in this approach.nuttx_main.cis reworked as well, as that is, posix-speaking, the standard, but still, it does not work really well.I'm going to submit a change that should make LinuxRT codegen possible, in nearest hours.
Documentation will be added soon.