Errors with Keysight Connection Expert

[raphaelvalentin]

Dear Coburnw,

I have a strong interest about this library on purpose to develop my instrument with a VXI11 protocol (understand code and convert it in ARM C++ for a MCU).
The library seems create errors when using with Keysight Connection Expert (C) (free to use).
I have seen that 'handle_19' is experiencing an issue, but also 'device_lock' and 'device_unlock'.

• The 'handle_19' seems having a problem with what KCE is sending. I am not sure if NI-VISA send different arguments?
• 'device_lock' and 'device_unlock' lead to another issue when these functions return ERR_OPERATION_NOT_SUPPORTED. If lock is not implemented, maybe it should be better to return ERR_NO_ERROR, I am not sure...

When I (bad) quickfix these issues, KCE is okay.
Thank you for your support.
Raphael.

[coburnw]

First off, quite a few bug fixes from ulda have been merged in. you should probobly pull in the latest.

I have never knowingly used the lock functions in any vxi-11 device, Im not sure where to start to debug it. I do see that table B.14 of the VXI-11spec does not show a OperationNotSupported option, so you certainly found something that should be addressed. But maybe the real problem is that the lock functionality is not optional?

Did you use time_device.py from the demo_servers folder for you boilerplate? If so, I suspect you could just override device_lock and device_unlock in the TimeDevice (YourDevice) before you attach it with add_device_handler. Your overrides would simply return err_no_err.

Are you thinking you might need device locking in your application?

[raphaelvalentin]

First off, quite a few bug fixes from ulda have been merged in. you should probobly pull in the latest.

I will have a look asap, thanks.

I have never knowingly used the lock functions in any vxi-11 device, Im not sure where to start to debug it. I do see that table B.14 of the VXI-11spec does not show a OperationNotSupported option, so you certainly found something that should be addressed. But maybe the real problem is that the lock functionality is not optional?

You got my question. Despite specifications or unmentioned data (which could let suggest that it can be optional), my thinking is to keep the compatibility with tools written by the majors. This is only my concern :-)

Did you use time_device.py from the demo_servers folder for you boilerplate? If so, I suspect you could just override device_lock and device_unlock in the TimeDevice (YourDevice) before you attach it with add_device_handler. Your overrides would simply return err_no_err.

Sure.

Are you thinking you might need device locking in your application?

At least on my Arm-based MCU instrument, some (non-blocking?) hardware operations need some time to be executed. "Lock" system may also provide information to a controller trying to initiate communication (with hardware control commands) that a hardware operation (or a list of operations) is currently pending, i.e., the command queue buffer is not ready to accept more commands, and it requires to wait a bit (concurrently with the busy bit). Despite this case, I need to cover possible multiple instrument connections due to the situation that I am limited in the number of sockets at the hardware level (I do no have an OS to manage such case)
But this is just a rough idea for the moment and I am still on the understanding of the protocol...

[ulda]

I have a commit not upstreamed yet for fixing unlock, see
cba578a

[coburnw]

I added a 'lock' branch with a prototype locking mechanism which you might try. As yet there is no lock on the lock so It will be susceptible to race conditions, and will return immediately on failure regardless of any timeout supplied. I would like to hear how it plays with Connection Expert, and if it works like one would expect.

Edit: Full locking support including timeouts has been rolled into the master branch. Locking is now fully handled by the library. I thought of adding a lock/unlock event, but couldnt really think of a use case. If you come up with a good reason, lets add it.

[raphaelvalentin]

Dear Coburn,

After installing rpcbind on cygwin, I run the script example time_device.py (from last github version) on cygwin, and I confirm that Keysight Connection Expert 2021 (Keysight VISA, available free cost from Keysight website) recognized the pseudo instrument without any errors. I will try in the future with ni-visa.
For some of us, this will open the inclusion of a featured raspberry pi as an instance of a test & measurement setup controlled so far with a vendor automation tool such as labview, iccap, or alternatives.
Thanks a lot for the update !!

Edit:
In instrument_device.py, class DefaultInstrumentDevice, method device_write, it may be appropriate to compare cmd.rstrip() == '*IDN?' and not cmd == '*IDN?', since usually a SCPI command requires a termination character ('\n', or '\r\n'). Same for '*DEVICE_LIST?'.

Sincerely,
Raphael.

[raphaelvalentin]

Dear Coburn,
I continue my test implementation with the vendor automation tool. I got stop here:

$ python ivi_driverfake.py
Press Ctrl+C to exit
INFO:__main__:starting time_device
DEBUG:vxi11_server.instrument_server:DeviceLock:__init__,l81 inst0
INFO:vxi11_server.instrument_server:abortServer started...
INFO:vxi11_server.rpc:registering (395183, 1, 6, 60443) on ('0.0.0.0', 60443)
INFO:vxi11_server.instrument_server:coreServer started...
INFO:vxi11_server.rpc:strting new request handler
DEBUG:vxi11_server.instrument_server:****************************
DEBUG:vxi11_server.instrument_server:CREATE_LINK (0, False, 0, 'inst0')
DEBUG:vxi11_server.instrument_server:Device name "inst0"
DEBUG:vxi11_server.instrument_server:handle_10,l273 self.link_id = 201
DEBUG:vxi11_server.instrument_server:DeviceLock:__init__,l81 XXX_1
INFO:vxi11_server.rpc:strting new request handler
DEBUG:vxi11_server.instrument_server:DEVICE_CLEAR (201, 8, 2000, 2000)
DEBUG:vxi11_server.instrument_server:DeviceLock:__call__,l125 args = (201, 8, 2000)
DEBUG:vxi11_server.instrument_server:DeviceLock:__call__,l129 error = 0
DEBUG:vxi11_server.instrument_server:****************************
DEBUG:vxi11_server.instrument_server:CREATE_LINK (0, False, 0, 'inst0')
DEBUG:vxi11_server.instrument_server:Device name "inst0"
DEBUG:vxi11_server.instrument_server:handle_10,l273 self.link_id = 202
DEBUG:vxi11_server.instrument_server:DeviceLock:__init__,l81 XXX_1
DEBUG:vxi11_server.instrument_server:DEVICE_CLEAR (201, 8, 2000, 2000)
DEBUG:vxi11_server.instrument_server:DEVICE_CLEAR (201, 8, 2000, 2000)
DEBUG:vxi11_server.instrument_server:DEVICE_LOCK (201, 9, 120000)
DEBUG:vxi11_server.instrument_server:handle_18,l460: self.link_id = 202
DEBUG:vxi11_server.instrument_server:handle_18,l465: error = 4

With a main script which includes a lock and looks like:

class FakeDevice(Vxi11.InstrumentDevice):

    def device_init(self):
        "Set the devices idn string etc here.  Called immediately after instance creation."
        self.idn = 'XXX INSTRUMENTS', 'MODEL XXX', '04476671', '1.6.7c'
        self.result = 'empty'
        self.lock = DeviceLock("XXX_1")
        return

    def device_write(self, opaque_data, flags, io_timeout):
        error = vxi11.ERR_NO_ERROR

        #opaque_data is a bytes array, so decode it correctly
        cmd=opaque_data.decode("ascii")

        if cmd.strip() == '*IDN?':
            mfg, model, sn, fw = self.idn
            self.result = "{},{},{},{}".format(mfg, model, sn, fw)

...

I do not control the flow of commands... From what I could understand, the vendor tool calls twice CREATE_LINKS which induces a change in the link_id. The vendor tool returns basically the error: "could not lock Available connection".

Sincerely,
Raphael.

[ulda]

hi Raphael
if i read cowburns code correctly, the device lock is created automatically while registering the device so that all device instances share the same lock
if you overwrite the lock in device_init, this is invalidated.
So try removing the line "self.lock=" in your script.

bye
ulf

[raphaelvalentin]

Dear Cowburns, dear ulf,

@ulf, you got right. And, unfortunately, it did not fix the issues.

However, I edit the file instrument_server.py as following and it seems that the vendor tool now not only recognize the connection but also recognize the fake device.

class Vxi11CoreHandler(Vxi11Handler):

    link_id = []                                                              # <---------

    def handle_10(self):
        '''The create_link RPC creates a new link.
        This link is identified on subsequent RPCs by the lid returned from the network instrument server.'''

        params = self.unpacker.unpack_create_link_parms()
        client_id, lock_device, lock_timeout, device_name = params

        logger.debug('****************************')
        logger.debug('CREATE_LINK %s' ,params)

        error = vxi11.ERR_NO_ERROR

        try:
            logger.debug('Device name "%s"', device_name)
            link_id, self.device = self.server.link_create(device_name)
            self.link_id.append(link_id)                                    # <---------
            logger.debug('handle_10,l273 self.link_id = %s', self.link_id)
        except KeyError:
            error = vxi11.ERR_DEVICE_NOT_ACCESSIBLE
            logger.debug("Create link failed")
        else:
            self.device.device_init()
            if lock_device == True:
                flags = 0
                error = self.device.lock.acquire(link_id, flags, lock_timeout)

        abort_port = 0
        if error == vxi11.ERR_NO_ERROR:
            abort_port = self.server.abort_port

        result = (error, link_id, abort_port, MAX_RECEIVE_SIZE)
        self.turn_around()
        self.packer.pack_create_link_resp(result)
        return

    def handle_23(self):
        "The destroy_link call is used to close the identified link.  The network instrument server recovers resources associated with the link"

        params = self.unpacker.unpack_device_link()
        link_id = params

        if link_id not in self.link_id:                                   # <---------

Basically, I did move the variable self.link_id into a class static variable and initialize as a list. List of handle methods have been changed accordingly (see method handle_23). Close connection method handle_26 clears the list. Destroy method handle_23 removes the link_id from the list.
This allows multiple link creations from a third party side to a device (as showed in my last post).
I am clearly not sure if it is under the vxi11 specifications but this trick works pretty well.

• What are your opinions?
• Also, does the lock shall be related to the link_id (local) or the device (global) ? I would believe that it shall be at device level. Edit: Maybe, I misunderstand :)

Below, find the log:

$ python ivi_fakedriver.py
Press Ctrl+C to exit
INFO:__main__:starting time_device
DEBUG:vxi11_server.instrument_server:DeviceLock:__init__,l81 inst0
INFO:vxi11_server.instrument_server:abortServer started...
INFO:vxi11_server.rpc:registering (395183, 1, 6, 53660) on ('0.0.0.0', 53660)
INFO:vxi11_server.instrument_server:coreServer started...
INFO:vxi11_server.rpc:strting new request handler
DEBUG:vxi11_server.instrument_server:****************************
DEBUG:vxi11_server.instrument_server:CREATE_LINK (0, False, 0, 'inst0')
DEBUG:vxi11_server.instrument_server:Device name "inst0"
DEBUG:vxi11_server.instrument_server:handle_10,l273 self.link_id = [201]
INFO:vxi11_server.rpc:strting new request handler
DEBUG:vxi11_server.instrument_server:DEVICE_CLEAR (201, 8, 2000, 2000)
DEBUG:vxi11_server.instrument_server:DeviceLock:__call__,l125 args = (201, 8, 2000)
DEBUG:vxi11_server.instrument_server:DeviceLock:__call__,l129 error = 0
DEBUG:vxi11_server.instrument_server:****************************
DEBUG:vxi11_server.instrument_server:CREATE_LINK (0, False, 0, 'inst0')
DEBUG:vxi11_server.instrument_server:Device name "inst0"
DEBUG:vxi11_server.instrument_server:handle_10,l273 self.link_id = [201, 202]
DEBUG:vxi11_server.instrument_server:DEVICE_CLEAR (201, 8, 2000, 2000)
DEBUG:vxi11_server.instrument_server:DeviceLock:__call__,l125 args = (201, 8, 2000)
DEBUG:vxi11_server.instrument_server:DeviceLock:__call__,l129 error = 0
DEBUG:vxi11_server.instrument_server:DEVICE_CLEAR (201, 8, 2000, 2000)
DEBUG:vxi11_server.instrument_server:DeviceLock:__call__,l125 args = (201, 8, 2000)
DEBUG:vxi11_server.instrument_server:DeviceLock:__call__,l129 error = 0
DEBUG:vxi11_server.instrument_server:DEVICE_LOCK (201, 9, 120000)
DEBUG:vxi11_server.instrument_server:handle_18,l460: self.link_id = [201, 202]
DEBUG:vxi11_server.instrument_server:locking device: inst0
DEBUG:vxi11_server.instrument_server:handle_18,l465: error = 0
DEBUG:vxi11_server.instrument_server:DEVICE_READSTB (201, 8, 100, 100)
DEBUG:vxi11_server.instrument_server:DeviceLock:__call__,l125 args = (201, 8, 100)
DEBUG:vxi11_server.instrument_server:DeviceLock:__call__,l129 error = 0
DEBUG:vxi11_server.instrument_server:DEVICE_READSTB (201, 8, 100, 100)
DEBUG:vxi11_server.instrument_server:DeviceLock:__call__,l125 args = (201, 8, 100)
DEBUG:vxi11_server.instrument_server:DeviceLock:__call__,l129 error = 0
DEBUG:vxi11_server.instrument_server:DEVICE_CLEAR (201, 8, 100, 100)
DEBUG:vxi11_server.instrument_server:DeviceLock:__call__,l125 args = (201, 8, 100)
DEBUG:vxi11_server.instrument_server:DeviceLock:__call__,l129 error = 0
DEBUG:vxi11_server.instrument_server:DEVICE_WRITE (201, 2900, 2900, 8, b'*IDN?\n')
DEBUG:vxi11_server.instrument_server:DeviceLock:__call__,l125 args = (201, 8, 2900)
DEBUG:vxi11_server.instrument_server:DeviceLock:__call__,l129 error = 0
INFO:__main__:inst0: device_write(): *IDN?
 XXXXINSTRUMENTS,MODEL XXXX,04476671,1.6.7c
DEBUG:vxi11_server.instrument_server:DEVICE_READ (201, 80, 2900, 2900, 0, 0)
DEBUG:vxi11_server.instrument_server:DeviceLock:__call__,l125 args = (201, 0, 2900)
DEBUG:vxi11_server.instrument_server:DeviceLock:__call__,l129 error = 0
DEBUG:vxi11_server.instrument_server:DEVICE_READSTB (201, 8, 2000, 2000)
DEBUG:vxi11_server.instrument_server:DeviceLock:__call__,l125 args = (201, 8, 2000)
DEBUG:vxi11_server.instrument_server:DeviceLock:__call__,l129 error = 0
DEBUG:vxi11_server.instrument_server:DEVICE_WRITE (201, 3000, 3000, 8, b'*IDN?')
DEBUG:vxi11_server.instrument_server:DeviceLock:__call__,l125 args = (201, 8, 3000)
DEBUG:vxi11_server.instrument_server:DeviceLock:__call__,l129 error = 0
INFO:__main__:inst0: device_write(): *IDN? XXXXXX INSTRUMENTS,MODEL XXXX,04476671,1.6.7c
DEBUG:vxi11_server.instrument_server:DEVICE_READ (201, 255, 3000, 3000, 0, 0)
DEBUG:vxi11_server.instrument_server:DeviceLock:__call__,l125 args = (201, 0, 3000)
DEBUG:vxi11_server.instrument_server:DeviceLock:__call__,l129 error = 0
DEBUG:vxi11_server.instrument_server:DEVICE_READSTB (201, 8, 3000, 3000)
DEBUG:vxi11_server.instrument_server:DeviceLock:__call__,l125 args = (201, 8, 3000)
DEBUG:vxi11_server.instrument_server:DeviceLock:__call__,l129 error = 0
DEBUG:vxi11_server.instrument_server:DEVICE_UNLOCK 201
DEBUG:vxi11_server.instrument_server:unlocking device: inst0

Thanks,
Raphael.

[ulda]

i did not read into details, but
this now looks very near to a behaviour according to spec of vxi-11-3, especially Figure B1
(download at http://www.vxibus.org/specifications.html )

[raphaelvalentin]

Dear Ulda,
@ulda, great!
Actually, with the Figure B1, the variable self.link_id may be not a static variable but may be attached in the method register of the class DeviceRegistry as following (the list is under a device_name):

    def register(self, name, device_class):
        if name is None:
            while 'inst' +  str(self._next_device_index) in self._registry:
                self._next_device_index += 1
            name = 'inst' +  str(self._next_device_index)

        if name in self._registry:
            raise KeyError

        item = DeviceItem(device_class)
        item.lock = DeviceLock(name)
        item.links_id = []                                       <----------

However, I still thinking that item.lock = DeviceLock(name) in the same method may be at wrong place and may be static (i.e. "fully global") (in the actual implementation, if I understand --not sure--, a RLock is created for each device_name, when it may be not dependant of a device_name also).

RULE B.3.3:The device_lock and device_unlock RPCs SHALL be implemented entirely within the TCP/IP-IEEE488.2 Instrument Interface. The TCP/IP-IEEE 488.2 Instrument Interface SHALL maintain locks as defined by the network instrument protocol

I will try this tomorrow.

Sincerely,
Raphael.

[coburnw]

Hi Raphael,

The item of note in your server's log is the duplicate device_lock calls. That can only fail as the lock can only be acquired once. Referring to the 'TCP/IP Instrument Protocol Specification VXI-11' Section B.4.2.Locking (page 19) states that a lock can only be acquired once. Rule B.6.72 (page 42) says that even if the client owns the lock, it should fail if it trys to acquire the lock again.

Are you are using the current master branch? Have you explored the locked_time_client.py sample in the examples? It interacts with the example time-server. If you run two instances of the client, the second will block for 10s or the first instance exits. restart the first and it will block until the second exits, and so on. Note the demo servers lack of interaction with the lock; the locking mechanism is invisible to the device (for better or for worse).

So first off I might suggest to you get the time_client.py, locked_time_client.py and time-server.py running on a clean clone then try again with your commercial clients. I worry that between the lack of documentation and your entering this project during a heavy state of flux is compounding your troubles. I suspect some success would feel good right about now, no matter how small.

Next, could you give me a big picture of the problem? Can you:

1. open the device without a lock then close?
2. open the device without a lock, acquire a lock, then close?
3. open the device with a lock then close?
4. open the device with a lock, acquire a lock, then close?

The expected result is number 4 fails.

Verify in the server log that when you open without any locking, that in fact no lock is being acquired. My sense is that there is an extra lock request happening somewhere in the shadows.

Finally, the https://github.com/python-ivi/python-vxi11 client library seems to lack the ability to lock the device during open. The locked_time_client imports its own repository's vxi11.py for its client library and if you modify the line

python-vxi11-server/vxi11_server/vxi11.py

Line 701 in 3d3c3e9

0, #lock_device,

changing the zero to 1 will force the open to request the device in the locked state. Note that this section of code is never used by the server, the change should affect the client only. (be sure to remember to back it out after fooling with it...)

Looking forward to your results,
Coburn

[coburnw]

Oh, and i wanted to thank you on your thoughts about stripping the newline from the scpi id string. I havent spent much time in the scpi spec. Ill dig into that soon.
c.

[raphaelvalentin]

Dear Coburn,
with my respect, I got a bit puzzled concerning the time_client.py, locked_time_client.py that you are suggesting. My testing configuration can be described as following:

• A python vxi11 server
• Keysight Connection Expert (KCE) (which corresponds to a GUI interface with a visa library).
• A third party software as a GUI client (which interacts with KCE -- could be NI-VISA --) and for which I do not control anything concerning the stack of commands (purely internal, a black box). I do not see any entries to get access to the low-level vxi11 commands (and so to plug a python client) which are handled I suppose via KCE (the visa function calls).

However, I agree that I am quite far to have digested entirely the xdr/rpc/vxi11 protocol documentations. My development time windows are shared with other activities and I am going step by step with different approaches.

My first idea is to fake an instrument (via the python vxi11 server), such the third party software get cheated and start to communicate with it seamlessly. Typically, it is possible because the result of the command *IDN? is used to identify an instrument. I consider this test case quite interesting as a study to understand exactly what is going on, and, hypothetically avoid the bias to have the description of a dedicated test client for a dedicated server (I assume KCE and the third party software very reliable). In other words, this could be called retro-engineering.

May I ask? In your ideas to check with the third party software, do you have an example of server scripts I could test straight using the last master branch? This will help me a lot and I will send you the log and any issues that can occur ;)

Thank you so much for your support,
Sincerely,
Raphael.

[coburnw]

I dont have a windows machine to attempt the keysight library for you. Could you give me a url to online documentation for the keysight gui you are using?

I think we should get back to basics by getting the python examples to work in a clean clone. Please take a look at this vxi11 client library https://github.com/python-ivi/python-vxi11 You can follow the instructions there to install it. That is the client that this repository was developed against so the example clients should work properly with the example devices. We can use that client instead of the keysight client until we get some confidence that the device is working in a way we expect.

Once you get the python example clients and servers running together, we can add the keysight app. All three should be able to work together. If the keysight app acquires the devices lock first, then the python client will fail, and visa-versa.

So how i can help you get the python examples running?
c.

[raphaelvalentin]

Dear Coburn,

I write an example of client/server based on a last git clone (see attached files). The client uses Royalty-Free Rohde-Schwarz visa library librsvisa.so interfaced by pyvisa (run on Ubuntu and can be downloaded at https://www.rohde-schwarz.com/us/applications/r-s-visa-application-note_56280-148812.html). From what I see, test4 and test5 are failing.

For my (only?) concern, the test4 reflects the third-party software communications, i.e. open 2 links for one instrument (e.g.instr0; see my previous log). However, is it part of the protocol?
Accessory, in visa, it appears some constants: AccessModes.no_lock / AccessModes.exclusive_lock / AccessModes.shared_lock. I am not sure to fully understand; does it apply at client-side or server-side?

What is your opinion about the unit test in regards to the vxi11 rules?

Thanks a lot for your support ;)

Raphael.

server.py.txt
test_client.py.txt