Request for source access or build customization (connection timeout BLE)

[Ruffus-ixo]

Hi Intercreate team,

I’m using smpmgr in production and would like to build a custom version with a longer BLE connection timeout (e.g. 30 seconds). However, it seems the public source code is missing internal modules such as smp and smpclient.

Could you please:

• Share the full source code (including those modules), or
• Provide a build with a configurable connection timeout?

Thanks in advance! Great tool!

[Ruffus-ixo]

Also it would be awesome to add an option to make the upgrade command to also confirm the image with image state-write in one go, not needing 2 separate ble connections, that would reduce DFU time when you are trying to update hundreds of devices through ble, idk if that's actually possible or makes any sense for you.

[JPHutchins]

I am curious if you are already using the upgrade convenience command or not:

│ upgrade    Upload a FW image, mark it for next boot, and reset  │
│            the device.

This abstracts the task completing a DFU + swap (if the system has multiple application slots).

Please describe your workflow a bit more so that I understand.

smpmgr image --help                              󰯸 smpmgr

 Usage: smpmgr image [OPTIONS] COMMAND [ARGS]...

 The SMP Image Management Group.

╭─ Commands ──────────────────────────────────────────────────────╮
│ state-read    Request to read the state of FW images on the SMP │
│               Server.                                           │
│ state-write   Request to write the state of FW images on the    │
│               SMP Server.                                       │
│ upload        Upload a FW image.                                │
╰─────────────────────────────────────────────────────────────────╯

smpmgr image state-write --help                  󰯸 smpmgr

 Usage: smpmgr image state-write [OPTIONS] HASH CONFIRM

 Request to write the state of FW images on the SMP Server.


╭─ Arguments ─────────────────────────────────────────────────────╮
│ *    hash         TEXT  SHA256 hash of the image header and     │
│                         body.                                   │
│                         [default: None]                         │
│                         [required]                              │
│ *    confirm            Confirm the image given by hash.        │
│                         [default: None]                         │
│                         [required]                              │
╰─────────────────────────────────────────────────────────────────╯

I think that you're talking about a sequence wherein you do:

• image upload - to the secondary slot
• image state-write - mark the uploaded image for "test".
• os reset

This is what the upgrade convenience command does, see the source here:

smpmgr/smpmgr/main.py

Lines 121 to 190 in 6651700

	@app.command()
	def upgrade(
	ctx: typer.Context,
	file: Annotated[Path, typer.Argument(help="Path to FW image")],
	slot: Annotated[int, typer.Option(help="The image slot to upload to")] = 0,
	) -> None:
	"""Upload a FW image, mark it for next boot, and reset the device."""
	try:
	image_info = ImageInfo.load_file(str(file))
	logger.info(str(image_info))
	except Exception as e:
	typer.echo(f"Inspection of FW image failed: {e}")
	raise typer.Exit(code=1)
	try:
	image_tlv_sha256 = image_info.get_tlv(IMAGE_TLV.SHA256)
	logger.info(f"IMAGE_TLV_SHA256: {image_tlv_sha256}")
	except TLVNotFound:
	typer.echo("Could not find IMAGE_TLV_SHA256 in image.")
	raise typer.Exit(code=1)
	options = cast(Options, ctx.obj)
	smpclient = get_smpclient(options)
	async def f() -> None:
	await connect_with_spinner(smpclient, options.timeout)
	with open(file, "rb") as f:
	await upload_with_progress_bar(smpclient, f, slot)
	if slot != 0:
	# mark the new image for testing (swap)
	image_states_response = await smp_request(
	smpclient,
	options,
	ImageStatesWrite(hash=image_tlv_sha256.value),
	"Marking uploaded image for test upgrade...",
	)
	if success(image_states_response):
	pass
	elif error(image_states_response):
	print(image_states_response)
	raise typer.Exit(code=1)
	else:
	assert_never(image_states_response)
	reset_response = await smp_request(smpclient, options, ResetWrite())
	if success(reset_response):
	pass
	elif error(reset_response):
	if error_v1(reset_response):
	if reset_response.rc != smperr.MGMT_ERR.EOK:
	print(reset_response)
	raise typer.Exit(code=1)
	elif error_v2(reset_response):
	if reset_response.err.rc != OS_MGMT_RET_RC.OK:
	print(reset_response)
	raise typer.Exit(code=1)
	else:
	assert_never(reset_response)
	else:
	assert_never(reset_response)
	print("Upgrade complete.")
	if slot != 0:
	print("The device may take a few minutes to complete FW swap.")
	asyncio.run(f())

See some notes about confirming here, and note that it should never been done prematurely: https://jphutchins.github.io/smp/latest/image_management/#smp.image_management.ImageStatesWriteRequest.confirm

The general approach to confirming the image is for the application to do so, thereby confirming non-brick. Of course it's safer to have the app not confirm itself after boot and rely on an SMPClient. So I think that a good solution here would be to add a --confirm option to update so that the full sequence is:

• image upload - to the secondary slot
• image state-write - mark the uploaded image for "test".
• os reset
• wait for device reconnect with some timeout - my personal experience is that reset swap confirmation and decryption can take more than a minute
• connect to the new app and send image state-write with the confirm flag

Re: factory, it should be noted that smpclient in general is capable of updating 10s, 100s, or 1000s of devices in parallel. Over UDP this is quite achievable, of course via USB & BLE there are some practical considerations for transport capability or simply the number of ports on the host. So, you might write your own factory application using smpclient - though, my selfish preference is to start work on parallel updates in smpmgr. The user interface for parallel updates would certainly need some thought, particularly around device discovery.

[JPHutchins]

I should also mention that there is a feature shell - which we need to rename since there's also shell SMP group - that I'd like to cleanup to allow a persistent transport function. Feel free to take a look! We'll probably rename it to "interactive" or similar.

Idea is you might do like:

smpmgr shell --ble my_device

And it would put the shell session in a context manager that keeps the transport layout open and handles disconnects/reconnects.

[JPHutchins]

Hi @Ruffus-ixo, thanks for the feedback!

Hi Intercreate team,

I’m using smpmgr in production and would like to build a custom version with a longer BLE connection timeout (e.g. 30 seconds). However, it seems the public source code is missing internal modules such as smp and smpclient.

Could you please:

• Share the full source code (including those modules), or
• Provide a build with a configurable connection timeout?

Thanks in advance! Great tool!

Here are links to the source code and licenses for our SMP libraries:

• smp (de/serialization) and pretty docs
• smpclient (transport implementations) and pretty docs

Note that when working on smpmgr, these packages will be brought into your local environment along with other dependencies, so you don't necessarily need to do anything with those repos directly: https://github.com/intercreate/smpmgr?tab=readme-ov-file#development-environment-setup

• Provide a build with a configurable connection timeout?

There is a bug here, and it's also poorly documented:

│ --timeout                 FLOAT              Transport timeout  │
│                                              in seconds; how    │
│                                              long to wait for   │
│                                              requests           │
│                                              [default: 2.0]     │

The --timeout arg is also used for the initial connection - so the first problem is that it needs to be documented in the --help output.

Notice that commands tend to use the connect_with_spinner convenience function, and they should pass the --timeout argument:

smpmgr/smpmgr/main.py

Line 147 in 6651700

await connect_with_spinner(smpclient, options.timeout)

The bug is that smpmgr is using wait_for, which was presumably implemented before smpclient added a timeout of its own. So, if the timeout given is longer than the default 5 seconds, the default 5 seconds times out first, effectively ignoring the longer requested timeout:

smpmgr/smpmgr/common.py

Lines 90 to 97 in 6651700

	try:
	await asyncio.wait_for(smpclient.connect(), timeout=timeout_s)
	progress.update(
	connect_task, description=f"{connect_task_description} OK", completed=True
	)
	return
	except asyncio.TimeoutError:
	logger.error("Transport error: connection timeout")

I've created #51 to track this bug.

It's not ideal that the single --timeout arg is used both for connection and request timeouts. This will be addressed by a rework of how transports are handled - allowing unique args to each transport - so I won't support a new --connection-timeout arg.

That work is being discussed here: intercreate/smpclient#58

[Ruffus-ixo]

Hi. Sorry for the delay. Been busy these days.
Basically i'm using a python script with a gui that generates temporary .bat files for every device to update :

set MAC={mac}
set BIN={bin_path}
set HASH={hash_str}

:RETRY_UPGRADE
echo Iniciando DFU upgrade para %MAC%
"{SMPMGR_PATH}" --timeout 120 --ble %MAC% upgrade "%BIN%" --slot 1
if errorlevel 1 (
echo Error en upgrade. Reintentando en 1 segundo...
timeout /t 1 >nul
goto RETRY_UPGRADE
)

echo Upgrade completado. Esperando 90 segundos...
timeout /t 90 >nul

:RETRY_CONFIRM
echo Confirmando firmware...
"{SMPMGR_PATH}" --timeout 120 --ble %MAC% image state-write %HASH% true
if errorlevel 1 (
echo Error en confirmación. Reintentando en 1 segundo...
timeout /t 1 >nul
goto RETRY_CONFIRM
)

echo Proceso finalizado correctamente.
echo ok > "{done}"
exit /b 0
""")

The first question I have is when the confirm command has changed from true -------> confirm
As you can see I'm using "brute force" by retrying until succes because of the ble connection timeout not working