Mark Rivers

I have written a test program to try to reproduce the error. The test program talks to 2 E-DIO24 modules using 4 threads. 2 of the threads talk to one E-DIO4, and the other 2 threads talk to the other module. There is a single mutex that the threads lock before they make the calls to the UL library. I can reproduce the problem in my EPICS driver ("Invalid network frame") if I don't lock the mutex before calling the UL library. My EPICS driver does have a mutex which is called when I thought it should be. But I just realized that mutex is only protecting access to a specific device, not to the UL library as a whole. So when there are 2 devices in the same EPICS process there are 2 mutexes, and the UL library could be called from 2 threads at the same time. Interestingly this does not appear to be a problem on Windows, and is not a problem on Linux with USB devices. It seems that the UL library on Linux for Ethernet devices cannot be called from 2 unsynchronized threads even if those threads are talking to different devices? testDualEthDevice.cpp

I just rebuilt uldaq with TRACE defined in ul_internal.h. When I now run my application controlling a single E-1608 I do not see any error messages. However, when I run it controlling 2 E-1608s I see the following error stream: [2023-May-01 10:57:50:873904] Invalid frame ID!!!! [2023-May-01 10:57:50:973725] Invalid frame ID!!!! [2023-May-01 10:57:51:074001] Invalid frame ID!!!! [2023-May-01 10:57:51:274613] Invalid frame ID!!!! [2023-May-01 10:57:51:374377] Invalid frame ID!!!! [2023-May-01 10:57:51:573951] Invalid frame ID!!!! [2023-May-01 10:57:51:674601] Invalid frame ID!!!! [2023-May-01 10:57:51:905766] Invalid frame ID!!!! [2023-May-01 10:57:52:073874] Invalid frame ID!!!! [2023-May-01 10:57:52:274010] Invalid frame ID!!!! [2023-May-01 10:57:52:474640] Invalid frame ID!!!! [2023-May-01 10:57:52:774197] Invalid frame ID!!!! [2023-May-01 10:57:52:874391] Invalid frame ID!!!! [2023-May-01 10:57:52:985038] Invalid frame ID!!!! [2023-May-01 10:57:53:173903] Invalid frame ID!!!! [2023-May-01 10:57:53:274781] Invalid frame ID!!!! [2023-May-01 10:57:53:374418] Invalid frame ID!!!! 2023/05/01 10:57:53.474 MultiFunction::readInt32 Error: Calling AIn, err=102 Invalid network frame 2023/05/01 10:57:53.474 dualTest:E1608_1:Ai3 devAsynInt32::processCallbackInput process read error [2023-May-01 10:57:53:474640] Invalid frame ID!!!! [2023-May-01 10:57:53:574945] Invalid frame ID!!!! [2023-May-01 10:57:53:674400] Invalid frame ID!!!! 2023/05/01 10:57:53.774 MultiFunction::readInt32 Error: Calling AIn, err=102 Invalid network frame 2023/05/01 10:57:53.774 dualTest:E1608_1:Ai4 devAsynInt32::processCallbackInput process read error [2023-May-01 10:57:53:774968] Invalid frame ID!!!! [2023-May-01 10:57:53:874470] Invalid frame ID!!!! [2023-May-01 10:57:53:975290] Invalid frame ID!!!! [2023-May-01 10:57:54:146314] Invalid frame ID!!!! [2023-May-01 10:57:54:475164] Invalid frame ID!!!! [2023-May-01 10:57:54:974780] Invalid frame ID!!!! [2023-May-01 10:57:55:074628] Invalid frame ID!!!! [2023-May-01 10:57:55:174388] Invalid frame ID!!!! [2023-May-01 10:57:55:474389] Invalid frame ID!!!! [2023-May-01 10:57:55:875013] Invalid frame ID!!!! [2023-May-01 10:57:55:974375] Invalid frame ID!!!! [2023-May-01 10:57:56:074441] Invalid frame ID!!!! 2023/05/01 10:57:56.174 MultiFunction::readInt32 Error: Calling AIn, err=102 Invalid network frame 2023/05/01 10:57:56.174 dualTest:E1608_2:Ai2 devAsynInt32::processCallbackInput process read error I now realize the following: - The error is Invalid frame ID. This is printed on this line in uldaq: https://github.com/mccdaq/uldaq/blob/1d8404159c0fb6d2665461b80acca5bbef5c610a/src/net/NetDaqDevice.cpp#L628 - The errors are happening 3-10 times per second. However, this line does 2 retries each time there is an error: https://github.com/mccdaq/uldaq/blob/1d8404159c0fb6d2665461b80acca5bbef5c610a/src/net/NetDaqDevice.cpp#L510 The retry generally works, so most of the time my application does not receive the 102 error. But clearly something is quite wrong because the uldaq driver is seeing all of these Invalid frame ID errors and is retrying 3-10 times per second. Thanks, Mark

Hi Fausto, I have now reproduced the problem on Ubuntu 22.04.1 LTS and Centos Stream 8. Thanks, Mark

Hi Fausto, I have not yet written a simple example program to demonstrate the problem, but I will do so. My real application is here: https://github.com/epics-modules/measComp/blob/master/measCompApp/src/drvMultiFunction.cpp It is part of the EPICS real-time control system toolkit used at many large-scale facilities worldwide, such as synchrotron and neutron sources, major telescopes, etc. The problem was first reported to me by a scientist at the Advanced Light Source at the Lawrence Berkeley Laboratory where he was using E-TC and TC-32 modules. I reproduced the problem at my facility, the Advanced Photon Source at Argonne National Laboratory, with 2 E-1608 units. I am using Centos 7 and Windows 10. Each OS is running on a dedicated machine, they are not running on the same system or on virtual machines. I have the ability to test on Ubuntu 22 and Centos 8 as well if you think that would be useful. The computers are using wired Ethernet on the same subnet as the Measurement Computing devices. As I said above, the problem does not occur on Windows, does not occur with multiple USB devices on Linux, and does not occur if the application is only running a single Ethernet device. The Linux version of my application previously used Warren Jasper's drivers, and I did not see the problem in that version. The problem only began when I switched to using uldaq rather than Warren's drivers. By the way I think uldaq is great! It is a clean design and remarkably problem-free for a new product. I really appreciate that you made it open-source on Github, I only wish that more vendors did the same! Thanks, Mark

I have now run the same application on Windows, controlling the same 2 E-1608 modules. The Windows application is the same source code as Linux, just using the equivalent calls to the Windows UL library rather than the uldaq library. In 20 hours there has not been a single error, while on Linux there would be about 100 errors in this time period.

I have discovered a problem when a single application (i.e. process) is controlling multiple Ethernet modules. The error message reported by my application is this: [Sat Apr 29 06:35:55 2023] 2023/04/29 06:35:55.680 MultiFunction::readInt32 Error: Calling AIn, err=102 Invalid network frame [Sat Apr 29 07:22:55 2023] 2023/04/29 07:22:55.840 MultiFunction::pollerThread Error: Calling CIn, err=102 Invalid network frame [Sat Apr 29 07:55:33 2023] 2023/04/29 07:55:33.678 MultiFunction::readInt32 Error: Calling AIn, err=102 Invalid network frame [Sat Apr 29 08:00:21 2023] 2023/04/29 08:00:21.678 MultiFunction::readInt32 Error: Calling AIn, err=102 Invalid network frame So the ulAin and ulCIn functions are returning error 102. The application is calling these functions many times per second. The errors are relatively rare, about once every 15 minutes on average. The problem has been observed when controlling multiple E-1608 modules, multiple E-TC modules, or multiple TC-32 modules. This error never occurs if the same application is only talking to a single Ethernet module. This error never occurs (nor any similar error) if the application is controlling multiple USB modules. I have run it with 3 USB modules for several weeks with no errors. My application runs on both Linux and Windows by using #ifdef __WIN32 or #ifdef linux to call either the Windows or Linux UL library. I will test to see if the problem also occurs when the application is running on Windows, but I don't think it does.

The USB-1808X data sheet says that the data FIFO for analog input is 4 kS, i.e. 4,096 samples. If the device is running at its maximum rate of 200 kHz on all 8 inputs, then the FIFO will fill in 4096/(200000*8) = 2.56 ms. It seems that would be the worst-case latency under those conditions, and it is probably under 1 ms if they are emptying the FIFO when it is less than half full.

Thanks, indeed changing to AI_CFG_OTD_MODE fixed that second issue.

I have run into 2 problems with the TC-32 on Linux. These problems do not occur on Windows with the equivalent function calls. The first problem is that the TC-32 contains 2 DIO ports. Port 0 is an 8-bit input port, and Port 1 is a 32-bit output port. On Windows this function queries the number of DIO ports: cbGetConfig(BOARDINFO, boardNum_, 0, BIDINUMDEVS, &numIOPorts_); It correctly returns 2. On Linux this function queries the number of DIO ports: ulDIOGetInfo(daqDeviceHandle_, DIO_INFO_NUM_PORTS, 0, &infoValue); It returns infoValue=4, not 2. My driver queries and prints the following information for each port. IOPort=0 DIO_INFO_PORT_TYPE=10, DIO_INFO_PORT_IO_TYPE=1 IOPort=1 DIO_INFO_PORT_TYPE=11, DIO_INFO_PORT_IO_TYPE=5 IOPort=2 DIO_INFO_PORT_TYPE=14, DIO_INFO_PORT_IO_TYPE=1 IOPort=3 DIO_INFO_PORT_TYPE=15, DIO_INFO_PORT_IO_TYPE=5 These are the definitions of the INFO_PORT_TYPE in uldaq.h: /** FirstPortA */ FIRSTPORTA = 10, /** FirstPortB */ FIRSTPORTB = 11, /** SecondPortA */ SECONDPORTA = 14, /** SecondPortB */ SECONDPORTB = 15, My driver has a poller thread that reads from all of the DIO ports reported when querying the number, providing that are not write-only ports. That is done with this call on Linux: status = ulDIn(daqDeviceHandle_, (DigitalPortType)digitalIOPort_[i], &data); The problem is that this returns error=50 for ports 2 and 3. 2023/04/26 16:54:38.151 MultiFunction::pollerThread Error: Calling DIn, err=50 Invalid port type specified 2023/04/26 16:54:38.152 portNumber=2 I suspect that the problem is that ports 2 and 3 are intended for use on the TC-32 expansion module. However, I need a way to determine if these ports really exist, i.e. if an expansion module is installed. The second problem is that on Linux, I get an error when setting open thermocouple detect. This error does not happen on Windows. This is my function: int MultiFunction::setOpenThermocoupleDetect(int addr, int value) { int status=0; static const char *functionName = "setOpenThermocoupleDetect"; if ((boardType_ != USB_TEMP) && (boardType_ != USB_TEMP_AI)) { #ifdef _WIN32 status = cbSetConfig(BOARDINFO, boardNum_, addr, BIDETECTOPENTC, value); #else OtdMode mode = value ? OTD_ENABLED : OTD_DISABLED; status = ulAISetConfig(daqDeviceHandle_, AI_CFG_CHAN_OTD_MODE, addr, mode); #endif reportError(status, functionName, "Setting thermocouple open detect mode"); } return status; } On Linux = ulAISetConfig is returning error 24. This is the report from my driver: 2023/04/26 15:48:47.764 MultiFunction::setOpenThermocoupleDetect Error: Setting thermocouple open detect mode, err=24 Configuration not supported The equivalent function call on Windows does not return an error. Thanks, Mark