Having Difficulty Running Multiple Waveforms SDK Instances at Once

[mowoodsiii]

Hello!

I've created a custom PCBA that allows a Raspberry Pi CM4 to be connected via a USB hub (TUSB2036) to 3x AD2s. The system works and I'm able to  identify and run example Waveforms SDK scripts using each of the AD2s individually. I can also open them up with the GUI version of Waveforms just fine.

However, when I run multiple instances of these Waveforms SDK example scripts simultaneously (using the `&` character in a parent BASH script to run my Python scripts in parallel in the background), the script(s) hang (I've included my script in this post). The following are worth mentioning:

1. When I kill the scripts, they print that the error occurs at the line which contains the following: `dwf.FDwfAnalogInStatus(hdwf, c_int(1), byref(sts))` (line 140 in attached). Again, just running a single instance of this code works fine; when running multiple, parallel instances, the code just hangs here, it doesn't crash.
2. It is worth noting that I can open 3x simultaneous instances of the Waveforms GUI, simultaneously, across my three separate AD2s. They will successfully open and run Oscilloscope VIs without error, so it doesn't seem to be a processing or USB issue...

Again, I've attached an abbreviated version of my modified SDK example code if that helps.

CM4AD2DAQ.py

[attila]

Hi @mowoodsiii

You could verity the functions return value, true/false 1/0 on success/failure
if dwf.FDwfAnalogInStatus(hdwf, c_int(1), byref(sts)) != 1:
            szerr = create_string_buffer(512)
            dwf.FDwfGetLastErrorMsg(szerr)
            print(szerr.value)
            quit()

You can specify the following after DeviceOpen to minimize device communication, so it is no real need to verify (eventually for invalid arguments) #Set functions but only #Config and #Status
dwf.FDwfDeviceAutoConfigureSet(hdwf, c_int(0)) # 0 = the device will only be configured when FDwf###Configure is called

[attila]

Hi @mowoodsiii

The WaveForms application uses the same API and your script seems to be running fine on a computer.

[attila]

Hi @mowoodsiii

Yes, I tested it with 3 devices. Earlier on wsl ubuntu and now on rpi4 ubuntu20 amd64.
I modified the script a bit to automatically open an available device, to record 2M samples (for longer capture/run) and removed file writing.
I don't know why it is failing for you...

[mowoodsiii]

Hi @attila! Thanks for the quick response!

It seems that the example code I pulled from the SDK samples already had invoked the `dwf.FDwfDeviceAutoConfigureSet(hdwf, c_int(0))` line, so that doesn't seem to be the issue...

I realize that I was being dense and wrote this reply while debugging the wrong file... I'll get back to this once I've actually played with the correct file   
(｡•́︿•̀｡) 

Edited February 16 by mowoodsiii

[mowoodsiii]

Okay! @attila et al, I've done some more careful poking,

It seems that there is some issue with having two (or more) SDK resources open and acquiring at the same time... I've noticed that things run fine as long as multiple devices run sequentially, but if they are simply open (using `dwf.FDwfDeviceOpen(####, byref(hdwf))` (lines 88 and 95)) while another device is running (even if the second device is just sitting idle after opening) , things start to hang.

Specifically, in my acquisition loop (lines 139 to 162) the device will report a `pcdDataAvailable` (named `cAvailable` in my code) value of 0 once another device is started (i.e. if a second device is started in the middle of the first device's acquisition loop, `cAvailable` will go from reporting non-zero values to zero values).

So maybe I'm misusing the device opening functionality? OR! Is there a chance that communication with the device is getting dropped because of too much simultaneous activity on the shared USB hub?

[mowoodsiii]

Interesting... (sorry to be so verbose... documenting my debugging/discovery process as a play-by-play)

I tried reducing the frequency of the acquisition (dramatically) and things worked fine. I reckon I'm sampling too fast and losing connection. But it's strange that I'm not getting some sort of error that the device is no longer a callable object...?

At first, I'd wonder if I'd expect to see a pop-up window if I weren't headless, but to reiterate about my setup: I've run three GUIs (non-headless) simultaneously with this setup without issue, so maybe it's specifically something to do with how I'm using the SDK?

*EDIT* Can confirm, I can successfully read from both oscilloscope channels on two devices simultaneously at 1MHz (way faster than the rate I have in my SDK code) using the GUI version of Waveforms with this same setup

Edited February 16 by mowoodsiii

[mowoodsiii]

Hmmm, interesting. Thanks @attila

I will say, the script works great on my end too, but to reiterate, I'm trying to run the attached script 3 times, simultaneously, in parallel. THAT is when I'm having issues...

[mowoodsiii]

Thanks for clarifying @attila! I appreciate your help! I'll keep digging on my end, likely an implementation issue

[mowoodsiii]

I suppose I have been digging into it more, and here's what I've found:
Ignoring running 3 simultaneous devices and focusing even on just the one, I run into a similar issue. Specifically:

• If my sampling rate is low (say, 100kHz... "sub-1MHz"), I seem to be able to collect as long of a capture as I want (2M samples, as you did @attila)
• If my sampling rate is bumped up to 1MHz, I can collect data fine if my sample size is less than 9160 samples (weird number...); I am left with a capture of 9160 samples 👍
• At that same 1MHz frequency, if I bump the sample size up to, say 9170, all of a sudden, I'll run into an infinite loop of `cAvailable=0`
  • Strangely, if I make it `break` once that's encountered and just write what I have, I always end up with 978 samples.
  • If I bump up the sample size to 20,000, now the `cAvailable=0` issue manifests after I've collected 3616 samples (which is way more that the 978 above) 
  • Similarly, for a sample cap of 200,000 samples, I hang at 159036 samples... (now with one sample flagged as `fCorrupted`)

Ugh...

It's ALWAYS consistent with these numbers. A given "frequency/sample size" pair of settings will always result in the same error at the same time, but I can't seem to see any pattern within it. And it's strange that, when, say, we use a higher sample size parameter, I end up with more successful samples taken before an error is encounter than when I have a smaller sample size...😓