ULx buffer size for input tasks

[banksey255]

The ULx for NI LabVIEW help states for the input task buffer size: "When you acquire a continuous number of samples (the ULx Timing VI's sample mode input is set to ContinuousSamples), ULx calculates the buffer size based on the number of channels, the sample rate, and the device's packet size requirements." How, precisely, is this calculated?

I ask as we are getting the following error despite calling the ULx Read VI approximately 50 to 100 times per second and asking it to read all the samples currently available in the buffer:

Error 29 occurred at ULx Read (Analog 2D DBL NChan NSamp).vi

Possible reason(s):

Measurements: Samples in the buffer were overwritten before they were read.

Reading the data more frequently, or specifying a fixed number of samples to read instead of reading all available samples might correct the problem.

 

In NI DAQmx, which the ULx driver appears to try to provide an identical interface to, it allocates a buffer at least as big as the value of the samples per channel attribute/property, and so in this respect it appears the ULx and DAQmx drivers differ in behaviour, and, based on the ULx driver help, I apparently have no control over the buffer size. Am I mistaken?

Edit: I managed to discover that the buffer size created by ULx for LabVIEW appears to have 1048576 elements, and since we are using a 10kHz sample rate with eight input channels, the buffer should be able to hold about 13 seconds of data, which makes me question whether that is the problem.

In case it is relevant, we are attempting to continuously stream from three USB-1808X at once using three separate tasks. To reiterate, we are reading each task 50 to 100 times per second. I came across another post where it was indicated that reading from multiple devices can be problematic: https://forum.digilent.com/topic/26880-how-do-i-get-two-analog-inputs-from-different-devices/#comment-81595. If so, is this only an issue with ULx for LabVIEW? That is, if I used LabVIEW to directly call UL for Windows or UL for .NET, then would this problem not occur? If so, which is higher performance, UL for Windows or UL for .NET? For example, it appears UL for .NET calls UL for Windows (CBW) under the hood, suggesting it would it be better if I call the latter from LabVIEW?

Edited November 8, 2023 by banksey255

[DAQman]

The ULx Timing VI Samples per Channel parameter is used to size the buffer, along with the device's packet size and the number of channels. I have three USB-1808 and can read them continuously at 50,000 S/s using the VI shown below. If you are getting an error indicating the buffer was overwritten, try increasing the Samples per Channel parameter.

 

[banksey255]

Hi JRys,

We are setting the Samples Per Channel parameter to the equivalent to 10 seconds worth of data whilst reading all data available in the buffer approximately 50 to 100 times per second (we do not rate limit the loop, but we do iterate on each ULx read until each returns something which is one thing that effectively limits the overall loop rate), and therefore it should be impossible to overrun due to an inadequate samples per channel value. In any case, as mentioned in my original post, according to the ULx for LabVIEW help, the Samples Per Channel has no effect on the buffer size when acquiring continuous samples, and I subsequently confirmed this behaviour.

One notable difference between your code and our code is that, when reading we use ULx Read (Analog 2D DBL NChan NSamp), we don't wire the number of samples per channel to read which causes it to default to -1 corresponding to reading all data available in the buffer, and we don't wire timeout as it is ignored when passing -1 for the number of samples per channel to read. In the way you have written your code, unless all devices run at the exact same clock speed, which will not occur as there is nothing synchronising the devices, you will eventually get an overrun error as samples will be getting acquired at slightly different rates by the different devices yet you read the same number of samples from each device per iteration causing the iteration rate to be limited by the slowest acquiring device, eventually causing the fastest acquiring device to overrun its buffer.

Edited November 8, 2023 by banksey255

[banksey255]

Hi JRys,

As part of my investigations, I restarted the LabVIEW application and found that the error no longer appears to occur (previously it appeared that we were getting this error message as soon as we started acquisition). This, of course, is not a satisfactory resolution as we do not know what the cause was, nor have any changes been made, therefore I presume it will recur at some point. I am communicating with end users to try to better understand what led to the error to see if a method to reproduce the error on demand can be identified.

[banksey255]

@JRys

I have investigated more. I want to highlight the following to help you diagnose the problem: If I take longer to read than what would take 1048576 values to be accumulated (which in our application corresponds to about 13 seconds worth of data), only then do I get error code 10017, which I find has an identical error message as error code 29 (see screenshot below). My suspicion is that the source of these errors is different, for example, error code 10017 is generated by the ULx LabVIEW driver, whereas perhaps error code 29 is generated by the hardware itself? Why would there be different error codes for the same thing?

Moreover, after error 10017, we can recover by stopping, clearing, recreating, and starting the task, however when we experienced error code 29, stopping, clearing, recreating, and starting the task resulted in more error code 29 as soon as we tried reading, despite reading within a fraction of a second, which is well below the 13 seconds worth of data that the buffer should be able to hold. Something must be fundamentally different between these two error codes: My question is, what is it?

I can see error code 29 appears to come from UL for Windows, however the help file effectively says this should never occur, unless the file input functions and methods are being used or if SINGLEIO is being used, and I presume the ULx LabVIEW driver does neither as it would be highly inefficient, but perhaps this suggests there is an issue in the ULx LabVIEW driver:

Edited November 13, 2023 by banksey255

[DAQman]

Our LabVIEW support is built upon an intermediate layer called ULx. Because no two MCC devices work the same, it attempts to present a unified front to LabVIEW to minimize VI code. It uses our Universal Library API, so underneath ULx is another layer communicating with the device driver. Unfortunately, documentation regarding error numbers greater than 1035 is not readily available. Most likely, it exists as comments in the code files. Error 29 bubbled up from the device, which means hardware FIFO overwrote unread data. The cure for this is to lower the sample rate.

When using SINGLEIO mode, error 29 is more common as it reads one scan of the channels at a time. It's useful for acquisitions with sample rates less than 1000 S/s.

I have a Dell desktop that seems to run my three-device test VI efficiently. It has an Intel i9-10920X 3.5 GHz processor,  64 GB RAM, an SSD HD, and six USB 3.0 ports. 

[banksey255]

@JRys

We have only experienced error 29 on one occasion, which required restarting our application to resolve (simply clearing and recreating tasks to restart acquisition without restarting the application continued to produce error 29), so when you tell me that you haven't experienced that error, it is not surprising.

Reducing the sample rate is not an acceptable solution and is a surprising suggestion since we are using a 10kHz sample rate, which is only 1/20th (5%) of the USB-1808X capability: Are you saying that these devices cannot even reliably operate at 5% of their rated ability? I would point out that our sampling of 8 channels at 10kHz corresponds to at most a data throughput of only 625 kB/s assuming the worst-case scenario of 64-bit values, which is miniscule. The USB-1808X uses USB 2.0, which we plug into either USB 2.0 or 3.0 ports, providing the full device bandwidth equal to at least 100 times what is required for our 10kHz sample rate. Therefore, there are no hardware bottlenecks (our DDR5 RAM and Intel Core i5-13500 CPU, of course, have far greater bandwidth than USB communications).

I guess the FIFO memory is on board the device? If hardware bandwidth between the device and the computer is not a limitation, what would cause the FIFO memory to overrun? Does this indicate faulty hardware, in which case we should return our USB-1808X for replacement?

Edited November 14, 2023 by banksey255

[DAQman]

Twenty-four channels at 10k S/s should be very much obtainable. I've attached my test vi for you to try. While running it, also launch the Windows Task Manager. Could you let me know the percentage of CPU usage? 

I don't believe your troubles are the result of faulty hardware. Every device that leaves the factory gets tested at full speed, and it's not a typical failure. 

The FIFO is small, holds up to 4000 samples, and is on the device. The driver is constantly checking for available data. Usually, when there is a FIFO error, it because the computer has background tasks running that prevent the driver from checking it.

 

3x USB_1808.vi

[banksey255]

@JRys

I will run your program at the first opportunity. Currently users are running my application that interfaces with the USB-1808X hardware and I will have to wait for them to finish. However, in the meantime, since it may be more useful anyway given the error was reported when using my application, here is a screenshot from device manager showing the resource usage when my application is running that shows the CPU usage is only 6%:

[banksey255]

@JRys

We have experienced error 29 again, and the currently reliability is unacceptable as the system we are commissioning will be required to acquire data for periods around a month. I have run your program and below is a screenshot of the task manager results (the CPU utilisation is approximately 15%). We need a solution.

[banksey255]

@JRys

As it appears to me that the CPU utilisation doesn't appear to reveal any issues, is there any way we can check the communication quality between the USB-1808X and the computer? Like a special debug version of the driver that you can provide that will report transmission errors? Basically, I'm wondering whether the signal-to-noise ratio in the communication between the USB-1808X devices and computer is so poor that there are lots of communication errors transferring the data causing transfer to slow significantly to the point where intermittently it isn't able to read the data off the devices quickly enough before being overwritten on the FIFO memory. I have no idea how to prove this with certainty unless the driver can report the number of communication errors with the devices, or if you instruct me in what to look for using USB sniffing software like Wireshark, etc.

[banksey255]

@JRys

Is there any new information on solving our error 29 problem?

[DAQman]

I have no solution at this time. I can run three devices, but l haven't performed a long-term test. Perhaps if it ran long enough, I would get the error.

If the USB-1808X cannot meet your requirements, check with your distributor about sending them back for a refund. 

[banksey255]

@JRys

Returning the devices would be our last resort as, in theory, they should be more than capable of meeting our requirements. I have even tested them acquiring at their maximum 200 kHz on all channels for a few hours without an error 29 occurrence.

You wrote that the USB-1808X FIFO holds up to 4000 samples: Is that the aggregate of all channels, or per channel? Are you sure about this value?

Continuing to investigate, I used Wireshark to assess the variation in the interval between subsequent USB BULK IN transfers (that is, the time interval between receiving data from the devices). Significant variation in this would indicate communication problems and is the only indicator I have as we do not have a hardware USB sniffer. The three plots below are for our three devices when simultaneously acquiring all 8 channels at 200 kHz: Is this amount of variation expected?

These intervals make no sense to me as I would expect the FIFO to either fill up to 20 ms if it was limited to 4000 samples per channel, or 2.5 ms if it was limited to 4000 samples all channels combined, both of which are exceeded by a significant number of the intervals, yet no error 29 occurred. The only two explanations I can come up with is either that the USB-1808X FIFO holds significantly more than the 4000 samples you mentioned, or the Wireshark timestamp is quite imprecise (but then still the average intervals were 4ms, 3.5ms, and 3.5ms for the three devices, respectively).

 

Edited November 24, 2023 by banksey255