Refactor to enable Driver commands for `/flex`, new Flex device support, testing via socat/VirtualDevice #162

yfyf · 2026-01-13T13:09:57Z

This PR materializes the idea of unifying Senso and Flex backends to have the same Driver command protocol, adds support for the new Sensitronics protocol/devices and revamps the testing workflow by introducing mocked serial devices (using socat).

Added functionality:

/flex endpoint WS clients can issue the same Driver commands as for the /senso backend. All commands except for firmware update are supported.
the Driver command protocol (see websocket/command.go) is updated to optionally include USB device metadata in DeviceInfo JSONs (in addition to discovered mDNS service metadata).
the Driver gains support for sending Broadcast messages to all connected endpoint clients. Currently /flex endpoint uses this to inform clients about change in device connectivity status. This eliminates the need to poll for changes from the client side. Not used in /senso.
A /flex client can now indicate it wants to manually control the device connection by passing an WS subprotocol manual-connect. This disables automatic scan+connect, requiring the client to manually issue Discover and Connect commands. Not used in Play, so can be thrown out, but this is a parity check that we can control Flex and Senso devices "in the same way" from Play.
Raw device serial data can now be recorded using tools/record-flex-serial (relies on strace, so Linux-only)
Driver can now be run in test mode (-test-mode flag), which enables vitrual/mock serial device registration via a POST /flex/mock request (used in tests and below)
tools/replay-flex now uses the Driver for replays and can replay both raw serial data (emulating a serial device via socat) and WS binary data (by asking the Driver to run with a "passthru" reader, see below)
Driver has two new Flex device (serial) readers: Sensitronics (for the new devices) and a special Passthru reader (for various dev/testing purposes)
Proper unit tests for the /flex endpoint
Added raw serial and WS data recordings for all current Flex devices

Changes:

go-serial is updated to a forked version that enables extra USB descriptors: bcdDevice, product and manufacturer. Tested to work on Linux and macOS. Windows support blindly implemented, not tested.
Driver used to (implicitly) sleep for 2 seconds on serial device disconnect/error. This is now removed, if a client aggressively attempts to reconnect without any backoff, so will the driver.
Avoid a temporary disconnect due to a race condition when two clients connect to the same /flex device at once

Not implemented, but considered (sorted by most->least important):

Driver does not send "derived" device metadata (e.g. deviceType=Flex, gen=v5) and so clients have to roll their own device identification logic. Mixed feelings about this. On the one hand, this is aligned with my (future) vision of the driver as merely a "dumb" chunking serial<->WS proxy, so that device firmware/protocol changes do not necessitate driver updates (e.g. if we add a new message type, the driver does not care). On the other hand, this requires duplicating the (USB -> device type/gen) identification logic in the client.
As before, Driver does not attempt to distinguish between Flex v4 and v5 and relies on client commands to switch between 8bit vs 12bit. This logic could be moved to the Driver in a backwards compatible way.
Recordings have no metadata, so replay-flex does not know what device to emulate, it has to be specified manually, which can be quite error prone
Recordings do not capture commands sent to the device (hence there's no way to know how the device was configured during the recording)
Would be good to replace USB device detection with udev or fsnotify on Linux at least, instead of manual sleep+polling. TBD in a separate PR.
the Discover commands remains awkwardly asynchronous in /flex (same as in in /senso) - the client has no way of knowing when the discovery is "finished", it can merely wait for the specified timeout to happen.
the Driver command protocol could be revised further and the implementation generalized/refactored even more, but this is PR is already too big.

Potential issues:

Windows support for the extra USB descriptors in go-serial is untested, as mentioned above
No way to record raw serial data in macOS, diminished dev experience. I see no easy/good way to achieve this, see Readme and 1d5cdf4 for extra details. Should not be a practical blocker, since for Sensitronics serial data ~= WS binary data.
Variance in sample timing in the serial recordings and replays, due to strace and more layers being involved. This can probably be adjusted by chunking the recording instead of replaying every read. I would say it's not a problem right now, since we don't rely on precise timing during replays.

Changes should be 100% backwards compatible for both /senso and /flex clients, i.e. Play should be able to run with this Driver "as is". Older Play versions will simply not be able to use Driver commands/events for Flex or understand the Sensitronics protocol.

Commits unsquashed, since they contain explanations about some of the choices. History is messy after the initial ~5 commits,so probably best to squash before merging.

Apologies for the huge diff.

Related PRs/code

Sensitronics firmware protocol: https://github.com/dividat/sensitronics-firmware/pull/4
Play implementation using /flex device metadata: https://github.com/yfyf/diviapps/tree/flex-discovery (fully working, but not yet rebased / cleaned up).

Checklist

Changelog updated
Code documented
Decide on a way to identify Flex/Sensitronics devices going forward. Currently Driver checks manufacturer == Sensitronics, which is maybe good enough, but need to set it in stone + document.

This is step 1 in introducing Driver commands to Flex: simply extracting the common (i.e. literally mostly identical) websocket handling code and encapsulating the device specific commands into a DeviceBackend interface. Adding some TODOs for trickier areas. Purposefully not introducing any new structure to keep the diff readable/minimal. At this stage this should be backwards compatible: Driver command handling is unchanged for the Senso, while Flex can now receive and respond to some of them, but keeps the backwards-compatible behaviour of auto-connecting on first websocket connection.

This adds proper Discover/Connect/Disconnect command handling to the Flex backend, while keeping the /flex endpoint backwards compatible with "auto-connect" behaviour. The auto-connect backwards compatibility is useful both for a seamless transition in Play and for allowing tools like `record-flex` to continue to "just work". By default, the Flex backend auto-connects to the first Flex-like serial device when a new WS connection is opened, as before. However, the client can pass the header `manual-connect: 1`, to indicate that it will set up the connection via Driver commands itself. If a previous client has already set up the connection, the header has no effect. Implementation change: this removes the 2 second sleep after serial connection closure (e.g. due to error). Blocking the thread seems like a strange way to implement backoff?

The fork provides extra USB device details on Linux. The go version bump is needed for basic generics support. The rest is due to `go mod tidy`

This extends the Driver to: - Support Flex device metadata in Driver message replies - Enable broadcasting messages to all connected clients Using it, the Flex backend implements a flow where it auto-connects to devices and informs about device changes by broadcasting a Status update. This eliminates the need for the client to periodically poll the Driver. The protocol changes are backwards compatible for the /senso endpoint.

- Handle port closure in the same context where it is opened - Move context cancelation to the caller - Remove explicit reader cancelation - it will automatically canceled when parent context is canceled

Even if device connection is not established, we still need to cancel the auto-connect if there are no subscribers remaining. Note: Disconnect already internally checks if `cancelCurrentConnection != nil`

Bumping node_modules/ws to be able to identify binary messages.

Also split up tests for checking different aspects.

The strace recording approach is a bit hacky, but the alternatives are even messier. If we want to both interact and observe, we have two choices: - multiplex the serial stream before it reaches the driver. This would mean circumventing the serial enumeration, registering a mock device just to record it, etc. Messy. - multiplex the serial stream once it reaches the driver, i.e. expose yet another binary stream from the driver. Complex and potentially dangerous code to have in production. Philosophical digression: with the new Flex protocol, Driver really becomes just a glorified chunker + WS<->USB/serial proxy. TODO: store metadata or at least a file suffix convention (, .v4.ws.dat, .v5.serial.dat) to help distinguish what type of data was recorded. Disclaimer: partially vibe-coded

yfyf · 2026-01-19T12:17:04Z

Changed to dividat fork, but will need a version/ref bump once dividat/go-serial#1 is merged.

EDIT: bumped.

~~Intention is to drop Windows support entirely, TBD in a separate PR.~~

knuton

Very good work, still working through it, but submitting some feedback mostly about code structure.

Will perform manual testing with physical Senso next.

src/dividat-driver/server/main.go

knuton · 2026-01-20T17:17:47Z

src/dividat-driver/protocol/main.go

+	} else if message.Discovered != nil {
+		serviceEntry := message.Discovered.TcpDeviceInfo
+		msg := struct {
+			Type string `json:"type"`


Should we consider adding an additional device tag field that explicitly encodes which kind of device was discovered?

It took me a while to understand what's happening here.

Maybe branch on which of TCP/USB is defined and don't "mix" definitions, even if a little more verbose?

👍 for adding the extra device tag that the client can use to simplify parsing.

But not entirely sure how you propose to branch/not "mix", while still being backwards compatible.

Currently Play receives Discover replies (for the Senso) that look like:

{ "type: "Discovered", "service": <TcpDeviceInfo>, "ip": [ "1.1.1.1", ... ] }

do you propose to re-use the service field, i.e.:

{ "type: "Discovered", "device": "senso", "service": <TcpDeviceInfo>, "ip": [ "1.1.1.1", ... ] } { "type: "Discovered", "device": "flex", "service": <UsbDeviceInfo> }

This would still "mix" things, as e.g. the IP is present for Senso, but not Flex devices.

Or do you mean moving service/ip to a sub-field, i.e. making things not backwards compatible? I really wanted to avoid having to make any kind of changes to the Senso part in Play to minimize the surface. I think it's better to tackle these rougher corners if/once we attempt to fully unify the device backends.

I was thinking to keep the format backwards compatible and flat as you made it, but use the device tag to make explicit what it should be parsed as.

No device tag: Implicitly parse only for TCP

device = "senso": Parse only for TCP

device = "flex" (or device = "serial"?): Parse only for USB

If there are superfluous fields for a given device tag, they can simply be ignored (this is the default behaviour in most JSON parsing).

I went a slightly different way and duplicated the DeviceInfo into a separate device field of the Status reply. Otherwise, it would mean we have two different serialization paths and hence two different parsing paths for Status vs Discovered messages. Also enforced safe/correct creation of DeviceInfo for the different device types.

Re naming: went for flex instead of serial, since otherwise it's a strange taxonomical mix in the enum ("senso" and "serial").

More details in the commit: ec74088

P.S. This breaks the PoC implementation in Play, will fix it after I'm done with other parts.

This is OK for me, though if I understand correctly could also be resolved by giving the Status message the same structure as discussed above for Discovered. Overall that would be less duplication, and "just" an implicit default for device type.

I might miss something because I have spent less time thinking about it.

by giving the Status message the same structure as discussed above for Discovered

Hmm, not sure I understand. Are you proposing to make JSON(Status) == JSON(Discovered)? This is possible, but would require even more duplicated fields for backwards compatibility (ip, address and service). Also, Status might include other fields in the future, so making Status == Discovered is probably not a good idea.

knuton · 2026-01-20T17:22:08Z

src/dividat-driver/websocket/main.go

Nit: "util" is underselling the importance of these modules, and util/websocket is very vague. "device"/"devices" or "protocol"/"client protocol"?

True, was thinking about this as well. Maybe transport/websocket? Or.. just websocket/?

Something emblematic of why I think websocket is not a very helpful name is code like this:

func (backend *DeviceBackend) GetStatus() websocket.Status { return websocket.Status{ Address: backend.address, } }

This is really not a WebSocket status at all, and also not a WebSocket address. So one needs to override intuitive reading with background knowledge completely in this function.

Right. I think the protocol part deserves it's own namespace, so maybe it can even become two modules.

Converted websocket/command.go to a protocol module (8e31859) and moved websocket to the top-level (2eab988). At first I did transport/websocket, but I don't think it makes much sense to have another layer since 1) there's only one transport currently; 2) the module does more than just handle transport; and 3) the top-level is anyway messy with modules that should not probably be top-level (e.g. service, logging and firmware).

src/dividat-driver/util/websocket/main.go

src/dividat-driver/flex/device/sensitronics/main.go

src/dividat-driver/flex/enumerator/main.go

src/dividat-driver/flex/main.go

knuton · 2026-01-22T11:01:51Z

src/dividat-driver/flex/main.go

+	reader := deviceToReader(device)
+	// should not happen
+	if reader == nil {


I might have missed something, but to me it looks like the enumerator only gave us a list of all Teensys, so this could very well happen?

Could we "parse, don't validate" and look up the reader earlier when checking for supported devices?

Yes, good point. Will maybe just move the reader-choice closer to enumeration, but need to think a bit.

I introduced a DeviceFamily identification stage and pushed to the enumerator, see: 83b5fdd

This does not allow me to get rid of the if reader == nil, but at least the comment is correct now. I don't see how to get rid of it without making enumerator directly return the reader instance, which I would find a bit odd, because it couples the identification with the specific parsing implementation. Maybe you see a better way, let me know.

Related to this change, we discussed whether to include the derived DeviceFamily type field in the UsbDeviceInfo or the Status reply, so that the client can know what the Driver chose as the parser. I am a bit reluctant to do it, because it would break the PASSTHRU concealment. I.e. if I say DeviceFamily: Passthru, the client can get confused or, worse, handle it in a "custom" way. This is of course only a dev/testing concern, so maybe I am overthinking it.

Readme.md

This modifies the protocol to serialize DeviceInfo in both Status and Discovered messages in an identical way. This simplifies the parsing logic for the clients, since they can: 1) Rely on the `deviceType` field to determine what fields are present 2) Re-use the same parsing logic in Status and Discovered messages To prevent incorrectly constructed DeviceInfo structs, all the fields are made private and helper functions are provided for initialization. To maintain backwards compatibility, Senso device data is duplicated in the top-level fields in Status messages. Since this data is used at a low-volume, this should not pose any problems.

This ensures that devices returned by ListMatchingDevices() are all supported devices that we know how to connect to. Previously, the enumerator would only check the Teensy Vendor ID. Also adjusted the tests and extended to verify unknown devices are not Discovered. Note: purposefully not providing DeviceFamily to the client, since it would break the "passthru" concealment.

src/dividat-driver/flex/enumerator/main.go

yfyf · 2026-01-26T08:31:39Z

Off-topic: the "CORS and PNA test" is flaky: https://github.com/dividat/driver/actions/runs/21350862315/attempts/1

I think I've seen it fail a long time ago too.

yfyf · 2026-01-26T08:33:35Z

@knuton addressed all review comments as of now. Let me know if you want me to patch Play side for testing. I'd prefer to wait for feedback, finalize here, then return to Play side.

knuton

Very good!

I confirmed compatibility with an unaltered main-branch version of Play, including discovery, commands, firmware update.

I was also able to replay v5 recordings via passthru with the same version of Play.

Has flex replay also been tested on macOS?

src/dividat-driver/flex/enumerator/mock_devices.go

src/dividat-driver/senso/main.go

src/dividat-driver/flex/main.go

knuton · 2026-01-27T13:11:14Z

src/dividat-driver/protocol/main.go

+	} else if message.Discovered != nil {
+		serviceEntry := message.Discovered.TcpDeviceInfo
+		msg := struct {
+			Type string `json:"type"`


This is OK for me, though if I understand correctly could also be resolved by giving the Status message the same structure as discussed above for Discovered. Overall that would be less duplication, and "just" an implicit default for device type.

I might miss something because I have spent less time thinking about it.

knuton · 2026-01-27T16:04:30Z

test/flex/mock/VirtualDevice.js

@@ -0,0 +1,143 @@
+// Disclaimer: this is 90% vibe-coded


OK, I agree that testing tools are under different constraints than the production code.

I am not sure what to practically make of the comment going forward -- is it somehow like generated code, do I need to keep vibing when I want to edit it? Probably not? Origin is good to document (commit message?), but from now on it's just code.

test/flex/mock/VirtualDevice.js

test/flex/mock/VirtualSerialPort.js

knuton · 2026-01-27T16:19:25Z

test/flex/mock/VirtualSerialPort.js

We don't really have codified JS code style in this repo, and you are adding a much more recent dialect than most of the existing files are written in (which is fine).

But maybe just drop semicolons, which is most obvious stylistic difference?

Dropping the semicolons actually breaks a bunch of tests in mysterious ways. Is it really a good idea to default to no semicolons?

tools/replay-flex/index.js

yfyf · 2026-01-28T09:04:28Z

Has flex replay also been tested on macOS?

No yet, should be done. Is make test known to work on macOS? Could add a test workflow for macOS in the CI.

Dropping the `sensitronics` alias for the `v6`, but keeping the possibility to identify the prototype devices when Manufacturer == Sensitronics.

Since each DeviceBackend uses a separate broker instances, there is no need to use globally unique names.

This ensures that samples are produced with the same delays they were recorded (previously it was shifted-by-one).

Dev builds use a `debug` tag by default. When the build is tagged with `debug`, a functional MockDeviceRegistry is created. Otherwise, a noop stub is used.

yfyf · 2026-01-28T14:23:51Z

Has flex replay also been tested on macOS?

No yet, should be done. Is make test known to work on macOS? Could add a test workflow for macOS in the CI.

I changed CI to run the tests on macOS and they seem to pass for Flex, but one Senso test fails (or is flaky):
https://github.com/yfyf/driver/actions/runs/21441751480/job/61746447633

EDIT: it's the "Can discover mock Senso" test, so seems to be related to mDNS/bonjour, could be CI specific. Removing the CI change for now, we can add it / fix it in a separate PR.

EDIT 2: confirmed by @terezka to work on her dev macOS machine locally, so it seems to be a CI / GH runner thing.

yfyf added 30 commits January 12, 2026 13:31

Move Driver protocol to separate file

484aa17

Bump go and replace serial package to forked one

83a5724

The fork provides extra USB device details on Linux. The go version bump is needed for basic generics support. The rest is due to `go mod tidy`

Move serial chunking outside and add Sensitronics support

d8caf08

Add bcdDevice to UsbDeviceInfo for identifying V4 vs V5 Flex

93cbe82

Remove unused const

08d87b8

Simplify context/cancelation management in ReadFromSerial

03acce7

- Handle port closure in the same context where it is opened - Move context cancelation to the caller - Remove explicit reader cancelation - it will automatically canceled when parent context is canceled

Ensure auto-connect is always disabled

7a5d931

Even if device connection is not established, we still need to cancel the auto-connect if there are no subscribers remaining. Note: Disconnect already internally checks if `cancelCurrentConnection != nil`

Move out Flex device enumeration to separate package

00290b4

Support creating/destroying mock devices via HTTP POST/DELETE

508fb2c

Announce testMode in main and louder

e7e4ce8

Fix: actually use the Broadcast messages

6f8df05

Add a passthru Flex device for testing purposes

f81271a

Add TODO note about manual-connect header

509c958

Initial Flex unit tests using Passthru

b653f7f

Bumping node_modules/ws to be able to identify binary messages.

Use Websocket subprotocol to control auto/manual-connect in Flex

114bed5

Also split up tests for checking different aspects.

Refactor test expects with helpers

d00d0b5

Upgrade broadcast test to multi-client

40e8d3d

Add a multi-device discover test (and fix a bug in impl!)

d884a2e

Enable changing replay speed in VirtualDevice

062b926

Upgrade replay-flex to use a mocked VirtualDevice

000a8ba

Replace minimist with commander for CLI parsing

4a4bb4a

Allow specifying the device type

8bbefa8

Move and rename v4 recordings

90027aa

Fixup: simplify TrimPrefix

458cc88

Add Flex v5 recordings

deed5f3

Add a basic test for v4

892d984

yfyf mentioned this pull request Jan 19, 2026

Eliminated unnecessary logs produced by Driver #163

Open

2 tasks

Use a tagged reference to dividat/go-serial

cdbab9c

knuton reviewed Jan 22, 2026

View reviewed changes

yfyf added 10 commits January 23, 2026 10:45

Name PASSTHRU- stripping more clearly

7b48814

Use a more idiomatic alias for serial/enumerator

544b17e

Use a more specific name for sendGeneric

66087b4

Use a more idiomatic alias for flex.enumerator

241ab6f

Move devices to a common module

4d0bd71

Move detailed Flex recording/replay docs to separate file

ffb149a

Move websocket/command.go to a 'protocol' module

8e31859

Make websocket a top-level module

2eab988

knuton reviewed Jan 26, 2026

View reviewed changes

src/dividat-driver/flex/enumerator/main.go Outdated Show resolved Hide resolved

Fix typo

eb763ce

knuton reviewed Jan 27, 2026

View reviewed changes

knuton added changes requested Changes to be made before next round of reviewing and removed reviewable Ready for initial or iterative review. labels Jan 27, 2026

yfyf added 6 commits January 28, 2026 11:24

Move vibe-coding disclaimers to git history

7481284

Identify V6 devices according to the agreed new USB descriptors

6b93fca

Dropping the `sensitronics` alias for the `v6`, but keeping the possibility to identify the prototype devices when Manufacturer == Sensitronics.

Drop flex- prefix from pubsub topic names

bf41097

Since each DeviceBackend uses a separate broker instances, there is no need to use globally unique names.

Sleep before writing sample data in replays

644d4e9

This ensures that samples are produced with the same delays they were recorded (previously it was shifted-by-one).

Replace runtime -test-mode flag with a 'debug' build tag

e9ca559

Dev builds use a `debug` tag by default. When the build is tagged with `debug`, a functional MockDeviceRegistry is created. Otherwise, a noop stub is used.

Use a better name for serial device family handlers

c91c8fd

yfyf force-pushed the flex-discover branch from 596ca23 to c91c8fd Compare January 28, 2026 15:00

Refactor to enable Driver commands for /flex, new Flex device support, testing via socat/VirtualDevice #162

Are you sure you want to change the base?

Refactor to enable Driver commands for /flex, new Flex device support, testing via socat/VirtualDevice #162

Uh oh!

Conversation

yfyf commented Jan 13, 2026

Related PRs/code

Checklist

Uh oh!

yfyf commented Jan 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

knuton left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yfyf Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

yfyf commented Jan 26, 2026

Uh oh!

yfyf commented Jan 26, 2026

Uh oh!

knuton left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

yfyf commented Jan 28, 2026

Uh oh!

yfyf commented Jan 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Refactor to enable Driver commands for `/flex`, new Flex device support, testing via socat/VirtualDevice #162

Refactor to enable Driver commands for `/flex`, new Flex device support, testing via socat/VirtualDevice #162

yfyf commented Jan 19, 2026 •

edited

Loading

yfyf Jan 23, 2026 •

edited

Loading

yfyf commented Jan 28, 2026 •

edited

Loading