Adding more tracking/events on the flashing and post install progress to help non-technical operators follow along the production process for mass production.

[MJDaBoss]

Hi,

I am trying to setup a manufacturing line that would be used to provision CM4s. We are currently still using bullseye on our CM4s and would like to continue using it for the time being. The CM4s will use a signed OS and secureboot for regular operation after provisioning is complete. Our team comprises of a bunch of electrical engineers and hence we lack PHP knowledge to do any development on the existing CMprovisioner.

The current process following stages:

• Flash
• encrypt
• verify device can boot in secure boot mode
• lock device

Our goal is simple:

• Mass produce a product based on CM4
• Setup a manufacturing line where an operator is non technical person.
• Design an interface for this operator who does not have any knowledge of the product.
• Simply show a status and a pass fail message.
• In case of a fail, recommend an action (retry provisioning, put aside for engineering inspection etc)
• Catch and contextualize different types of errors to help take the appropriate action (status codes to tell us what failed)

One of the things that we would like to do to help the operator is to give him simple feedback through UI, for that we need to be able to catch events and raise any additional events we might find useful.

I have following questions:

• Is there a way to track the status of TFTP file transfer as the image is sent over?
• How can we add additional events for?:
  • file transfer beginning
  • tracking file transfer progress.
  • file transfer completion
  • post install script download
  • post install script starting
• How can we capture these events in PHP or better yet another language like python since we lack PHP knowledge?
• Can we toggle a bunch of I/Os to track the progress of the provisioning stages (flashing start, flashing complete, encryption start (post-install script) encryption done, device locked etc.
• Can we add return codes of some kind and associate those to each stage either passing or failing, which can be mapped to a status code event?
• If we stick to PHP, can this be addressed by:
  • Touching the controller for scriptexecute?
  • Updating the blade php file for script execute to
  • creating additional custom events?
  • adding customizations to scriptexecute image?

I will appreciate some help and guidance on this.

Thanks.

[maxnet]

Is there a way to track the status of TFTP file transfer as the image is sent over?

Only a very tiny Linux utility operating system is sent over TFTP, and that is booted on the CM being provisioned.
That utility OS will download the final operating system over HTTP and write it to SD card while it downloads.

Can the progress of a HTTP download be tracked?
Not with the current setup in which the nginx webserver serves the image file directly as a static file to the requesting client.
As the free version of nginx does not offer a way to get proper progress information on running requests (its ngx_http_stub_status_module is very limited).

There do are other ways to do achieve that.
E.g. let PHP instead of nginx handle serving the image file and let that record progress somewhere (that do would require that nginx's buffering is disabled in its configuration, otherwise PHP will send the output to nginx's memory buffers instantly, and think the client is done downloading the file instantly).
Or switch to different webserver software that does allow the status of running downloads to be queried. lighttpd does have a mod_status module, where you can call a /server-status endpoint, and obtain the byte count on each individual running request, although that do would require that you add some logic to match the right request to the right client (e.g. based on remote IP, or telling the client to add an identifying query parameter to the URL when it downloads the file).

If you want more interactive progress reporting, broadcasting status updates using Laravel echo/websockets would be an option.
Do be aware that getting that right is easily a full day of work for an experienced PHP developer though.
And may not the best project to start with, if you haven't used it before...

[MJDaBoss]

@maxnet, First i really appreciate a quick response on this.

I am digging more and more in to the cmprovisioner and i think we can reduce the scope of things by letting go for the file transfer tracking.

I see a very daunting solution that involves scraping UART output from the CM4 IO board but it will not be very efficient.

Putting aside the file transfer tracking, could we approach things at the pre install and post install script level. For this, first I want to ensure my understanding of flow of the process is correct for a standard image is being written on a CM4 without any modification to the software. Is it correct to assume that:

• Device performs a network boot and receives a scriptexecute environment
• Device runs the scriptexecute based on cmdline and receives pre-install script (if any)
• Device completes pre-install(s) and starts image file transfer from the server and writes to the selected storage through dd
• Upon completion of transfer it receives post-install script(s)
• Device executes post_install script(s)
• Provisioning is completed

Here are some actions we perform and events captured by cmprovisioner:

Action	Captured in cmprovisioner	Notes
Provisioning start	Yes	- -
Image transfer begin	No	Could be synonymous to provisioning start if no pre-install script is used
Image transfer failed	Yes	--
Image transfer complete	No	--
Post install begin	No	This could also indicate completion of file transfer
Mount file system	No	Part of post install, failure sends a status code to server before exit or raise specific exit code
Check and resize filesystem to the storage	No	Part of post install, failure sends a status code to server before exit or raise specific exit code
Begin encryption	No	Part of post install, failure sends a status code to server before exit or raise specific exit code
Encryption complete	No	Part of post install, failure sends a status code to server before exit or raise specific exit code
Provisioning complete	Yes	--

On the dashboard view we see some of the logs like the provisioning start and completed or any failures if script exits with non-zero code. Could I extend that approach and capture more logs in that way. Cmprovisioner does shows a complete output of the scripts once the provisioning is completed.
What process currently gets that information from the CM4?

Assuming its provisioning start and complete are events, is it be possible for me to raise events for the actions I am performing in pre-install and post install scripts and extend the existing code to capture those?

Could I use curl -x post <<some URL on server>> to send an event with some parameters to identify an action being performed by the post install script?

[maxnet]

What process currently gets that information from the CM4?

Relies on the script running on the CM4 uploading the log output.
REST call on the client:

Good flow: https://github.com/raspberrypi/cmprovision/blob/main/resources/views/scriptexecute.blade.php#L54
Bad flow:
https://github.com/raspberrypi/cmprovision/blob/main/resources/views/scriptexecute.blade.php#L48-L52

Server side sticks it in the model/database: https://github.com/raspberrypi/cmprovision/blob/main/app/Http/Controllers/ScriptExecuteController.php#L268

Could I extend that approach and capture more logs in that way.

Capture, as in save to the database for debugging purposes if a problem arises yes.
But keep in mind that any output your custom script does already ends up in that log, so generally no need to extend anything, since it already has everything, provided your script does complete eventually (and does not hang in an endless loop).

Was not designed for real-time feedback to operators though.
Other technologies like Websockets are better suited for that...

Assuming its provisioning start and complete are events, is it be possible for me to raise events for the actions I am performing in pre-install and post install scripts and extend the existing code to capture those?

No.
Most are not implemented as events.

Only CmProvisioningComplete is a true Laravel event you can listen to in your own code without having to modify a single line of existing source code, and it is only emitted on success.
(Just toss your own listener code file in /var/lib/cmprovision/app/Listeners and you are done. Along the lines of: #45 ).

Everything else like the start of provisioning or error conditions were not designed to be hooked into, do not use Events/Publisher/Listener paradigm, and does require you to change core code if you want it to behave differently than it does now.

[gstrauss]

Or switch to different webserver software that does allow the status of running downloads to be queried. lighttpd does have a mod_status module, where you can call a /server-status endpoint, and obtain the byte count on each individual running request, although that do would require that you add some logic to match the right request to the right client (e.g. based on remote IP, or telling the client to add an identifying query parameter to the URL when it downloads the file).

lighttpd mod_magnet provides some sample Lua code for tracking upload progress, and the beauty of Lua is that the code is customizable and could be modified to report something else like download progress.