UnicodeEncodeError crash on Windows (cp1252) from Rich spinner characters

[brettlhartt]

Description

smpmgr upgrade crashes with UnicodeEncodeError on Windows when the console codepage is cp1252 (the default for most Western Windows systems). The Rich library's braille spinner character \u2819 cannot be encoded in cp1252, causing the entire upgrade to abort even though the upload may have started.

Steps to Reproduce

1. Windows 11 with default cp1252 console codepage
2. Run:

   py -m smpmgr --port COM14 upgrade firmware.signed.bin
   

3. smpmgr crashes during the progress spinner display

Expected Behavior

The upload should complete without crashing. The progress display should use characters compatible with the console encoding, or smpmgr should set PYTHONIOENCODING=utf-8 / handle encoding gracefully.

Actual Behavior

UnicodeEncodeError: 'charmap' codec can't encode character '\u2819' in position 0: character maps to <undefined>

Full traceback (trimmed):

  File "...\rich\_win32_console.py", line 441, in write_styled
    self.write_text(text)
  File "...\rich\_win32_console.py", line 402, in write_text
    self.write(text)
  File "...\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)
UnicodeEncodeError: 'charmap' codec can't encode character '\u2819' in position 0: character maps to <undefined>

The crash originates in Rich's LegacyWindowsTerm when rendering the braille spinner (\u2819).

Workaround

Setting PYTHONIOENCODING=utf-8 before running smpmgr avoids the crash, though spinner characters render as mojibake:

set PYTHONIOENCODING=utf-8
py -m smpmgr --port COM14 upgrade firmware.signed.bin

Environment

• OS: Windows 11 Pro 10.0.26200
• Python: 3.12.7
• smpmgr: 0.17.0
• smpclient: 6.1.0
• smp: 4.0.2
• Console codepage: cp1252 (Western European, Windows default)

[JPHutchins]

Thanks! I've never heard of console code page, what is that? Generally, it should work without any configuration in both flavors of PowerShell, and probably command prompt, but I would recommend either flavor of PowerShell over that.

Possible root cause: Textualize/rich#2882

What terminal are you using?

[JPHutchins]

Looks like PEP 686 will make this default in 3.15: https://peps.python.org/pep-0686/

[brettlhartt]

I've been using claude code to do some automatic PID tuning on our soldering irons, and it needs to re-load firmware regularly, so I don't fully understand why claude would fail running smpmgr in powershell when running smpmgr directly in powershell hadn't ever failed before. Here is the response claude provided when I asked it to answer your questions

The "console code page" is the Windows legacy encoding that Python uses for stdout by default — on most Western Windows systems
it's cp1252 (Windows-1252). You can check it with python -c "import sys; print(sys.stdout.encoding)" or chcp in cmd. It's
separate from the terminal itself — it's the encoding Python picks up from locale.getpreferredencoding().

I hit this running smpmgr from a Python subprocess (subprocess.run with capture_output=True) on Windows 11 with Command Prompt
as the underlying shell. But the encoding is cp1252 regardless of whether I use Command Prompt, PowerShell, or Windows Terminal
— it's the system-level ANSI code page, not a terminal setting. sys.stdout.encoding reports cp1252 in all of them on this
machine.

The Rich issue you linked (#2882) looks like the same root cause. Good news that PEP 686 will default to UTF-8 in 3.15 — that
should fix it for good. In the meantime, PYTHONIOENCODING=utf-8 works as a workaround on our end.

[JPHutchins]

OK, so it sounds like the fix is for that environment variable to get set in the subprocess API.

Since you're in a python program already, you can get a lot more flexibility by using smpclient directly. You could start by pointing Claude at these excerpts:

smpmgr/smpmgr/main.py

Lines 145 to 237 in d38b401

	@app.command()
	def upgrade(
	ctx: typer.Context,
	file: Annotated[Path, typer.Argument(help="Path to FW image")],
	slot: Annotated[int, typer.Option(help="The image slot to upload to")] = 0,
	confirm: Annotated[
	bool,
	typer.Option(
	"--confirm",
	help="Permanently confirm the image (prevent revert/rollback). "
	"Without this flag (default): the image is marked for test swap, and MCUboot will "
	"revert to the previous image if the new image fails to boot or is not confirmed. "
	"With this flag: the image is immediately marked as permanent, bypassing MCUboot's "
	"test/revert safety mechanism. "
	"[red]WARNING[/red]: Using --confirm skips the test boot and can brick your device. "
	"[bold]Only use this flag in development scenarios where recovery is possible[/bold], "
	"such as: MCUboot has serial recovery enabled and you have access to the physical "
	"serial interface; JTAG/SWD debug headers are exposed and not locked; "
	"or you have another reliable recovery mechanism. "
	"[red]DO NOT[/red] use --confirm in production or field deployments where physical "
	"access for recovery is not available. "
	"Best practice: always test first by marking for test swap, "
	"rebooting to verify the image works, "
	"then confirm the running image with 'smpmgr image state-write --confirm' "
	"(or some other mechanism).",
	),
	] = False,
	) -> None:
	"""Upload a FW image, mark it for next boot, and reset the device."""
	try:
	image_info = ImageInfo.load_file(str(file))
	logger.info(str(image_info))
	except Exception as e:
	typer.echo(f"Inspection of FW image failed: {e}")
	raise typer.Exit(code=1)
	try:
	image_tlv_sha256 = image_info.get_tlv(IMAGE_TLV.SHA256)
	logger.info(f"IMAGE_TLV_SHA256: {image_tlv_sha256}")
	except TLVNotFound:
	typer.echo("Could not find IMAGE_TLV_SHA256 in image.")
	raise typer.Exit(code=1)
	options = cast(Options, ctx.obj)
	smpclient = get_smpclient(options)
	async def f() -> None:
	await connect_with_spinner(smpclient, options.timeout)
	with open(file, "rb") as f:
	await upload_with_progress_bar(smpclient, f, slot)
	if slot != 0 or confirm:
	image_states_response = await smp_request(
	smpclient,
	options,
	ImageStatesWrite(hash=image_tlv_sha256.value, confirm=confirm),
	"Marking uploaded image for permanent upgrade..."
	if confirm
	else "Marking uploaded image for test upgrade...",
	)
	if success(image_states_response):
	pass
	elif error(image_states_response):
	print(image_states_response)
	raise typer.Exit(code=1)
	else:
	assert_never(image_states_response)
	reset_response = await smp_request(smpclient, options, ResetWrite())
	if success(reset_response):
	pass
	elif error(reset_response):
	if error_v1(reset_response):
	if reset_response.rc != smperr.MGMT_ERR.EOK:
	print(reset_response)
	raise typer.Exit(code=1)
	elif error_v2(reset_response):
	if reset_response.err.rc != OS_MGMT_RET_RC.OK:
	print(reset_response)
	raise typer.Exit(code=1)
	else:
	assert_never(reset_response)
	else:
	assert_never(reset_response)
	print("Upgrade complete.")
	if slot != 0:
	print("The device may take a few minutes to complete FW swap.")
	asyncio.run(f())

smpmgr/smpmgr/image_management.py

Lines 142 to 172 in d38b401

	async def upload_with_progress_bar(
	smpclient: SMPClient, file: typer.FileBinaryRead | BufferedReader, slot: int = 0
	) -> None:
	"""Animate a progress bar while uploading the FW image."""
	with Progress(
	TextColumn("[bold blue]{task.fields[filename]}", justify="right"),
	BarColumn(),
	"[progress.percentage]{task.percentage:>3.1f}%",
	"•",
	DownloadColumn(),
	"•",
	TransferSpeedColumn(),
	"•",
	TimeRemainingColumn(),
	) as progress:
	image = file.read()
	file.close()
	task = progress.add_task("Uploading", total=len(image), filename=file.name, start=True)
	try:
	async for offset in smpclient.upload(image, slot):
	progress.update(task, completed=offset)
	logger.info(f"Upload {offset=}")
	except SMPBadStartDelimiter as e:
	progress.stop()
	logger.info(f"Bad start delimiter: {e}")
	logger.error("Got an unexpected response, is the device an SMP server?")
	raise typer.Exit(code=1)
	except OSError as e:
	logger.error(f"Connection to device lost: {e.__class__.__name__} - {e}")
	raise typer.Exit(code=1)