Lost And Corrupted Data - Analog discovery Pro

[DroneGuru]

Hello,

We're trying to understand where does the lost and corrupted data flags come from during acquizition.

On this webpage https://digilent.com/reference/software/waveforms/waveforms-sdk/reference-manual, we can find :

Quote

 

FDwfAnalogInStatusRecord(HDWF hdwf, int *pcdDataAvailable, int *pcdDataLost, int *pcdDataCorrupt)

Desacription: Retrieves information about the recording process. The data loss occurs when the device acquisition is faster than the read process to PC. In this case, the device recording buffer is filled and data samples are overwritten. Corrupt samples indicate that the samples have been overwritten by the acquisition process during the previous read. In this case, try optimizing the loop process for faster execution or reduce the acquisition frequency or record length to be less than or equal to the device buffer size (record length ⇐ buffer size/frequency).

 

"The device recording buffer" : but according to this diagram, there is 2 buffers ;

The FPGA buffer, and the DDRam. So can you tell me which buffer you're talking about when you refer to "the device recording buffer" please ?

Thank you by advance,
Best regards.

Hugo

 

 

 

[attila]

Hi @DroneGuru

It refers to the FPGA buffer.

With DDR buffering make sure not to exceed the following limits for 1, 2 or 3/4 enabled channels.
Above 128Mi samples the rate will be limited by the USB or Ethernet, around 4MS/s depending on the system, network... 

 

In the newer software version some recording issues are fixed:

 

The installed pdf manual in the SDK folder in newer than the online variant and corresponds to the used software version:

[Andras]

Szia @attila!

Thank you for the explanation!

I would need your help. I have been writing my own C# tool to control my AD2 and capture data from its scope. It got better over the past months, but I still occasionally get corrupted data even at reasonably low rates, typically at 250kSPS. It looks like WaveForms has no trouble at all even at higher rates.

Here is below how my data acquisition loop looks like, could you take a quick look?

I use DWF v3.18.55, the loop thread runs with the highest priority, and I use AD2 with the 2x 16 Ki scope setting. The samplesBuffer is a shifting buffer implementation.

int i = 0;

while (!terminateAcquisition)
{
	while (true)
	{
		i++;

		if (i % 100 == 0)
		{
			if (availableBytes != totalBytes)
			{
				logger.LogWarning($"Data acquisition device reported some errors! Good: {(availableBytes / totalBytes).ToString("0.0%").PadLeft(6)} | Corrupted: {(corruptedBytes / totalBytes).ToString("0.0%").PadLeft(6)} | Lost: {(lostBytes / totalBytes).ToString("0.0%").PadLeft(6)}");
			}

			totalBytes = 0;
			availableBytes = 0;
			corruptedBytes = 0;
			lostBytes = 0;
		}

		Thread.Sleep(5);
		dwf.FDwfAnalogInStatus(dwfHandle, 1, out byte sts);

		if (!((samplesBuffer.TotalWrites == 0) && ((sts == dwf.DwfStateConfig) || (sts == dwf.DwfStatePrefill) || (sts == dwf.DwfStateArmed))))
			break;

		logger.LogWarning($"Data acquisition device got into an unusual state! sts:{sts}");
		Thread.Sleep(500);
	}

	dwf.FDwfAnalogInStatusRecord(dwfHandle, out int cAvailable, out int cLost, out int cCorrupted);

	if (cAvailable == 0)
	{
		logger.LogWarning($"Aqusition error! cAvailable: {cAvailable:N0}");
		Thread.Sleep(500);

		logger.LogTrace($"Reseting device...");
		dwfHandle = ResetDevice();

		bufferError = false;
		samplesBuffer.Clear();
		continue;
	}

	int cTotal = cAvailable + cLost + cCorrupted;
	if ((cLost > 0) || (cCorrupted > 0))
	{
		bufferError = true;
	}

	totalBytes += cTotal;
	availableBytes += cAvailable;
	lostBytes += cLost;
	corruptedBytes += cCorrupted;

	dwf.FDwfAnalogInStatusData(dwfHandle, 0, voltData, cAvailable);     // get CH1 data chunk

	if (bufferError)
	{
		samplesBuffer.Error(cAvailable, cLost, cCorrupted, (int)cTotal);
		bufferError = false;
		continue;
	}

	double[] voltDataAvailable = new double[cAvailable];
	Array.Copy(voltData, voltDataAvailable, Math.Min(cAvailable, voltData.Length));
	samplesBuffer.Write(Array.ConvertAll<double, float>(voltDataAvailable, v => (float)v));
}

If you don't mind sharing, I would be very interested in your data recording loop in WaveForms. Does it use DWF as well to communicate with the device?

Thank you,
Andras

[attila]

Szia @Andras

The record in the WF app is similar to the SDK/ samples/ py/ AnalogIn_Record_Trigger.py but it also includes mixed mode (analog+digital) recording.

The occasional data loss may be to some hiccup in the system, like other application or services blocking the computer (cpu,dma,bus...).
Try removing the Thread.Sleep(5);
EDIT: The cAvailable zero can be expected/accepted, specially at lower rates.

Edited November 3, 2022 by attila

[Andras]

Szia @attila,

Thanks for the tip, I'll check that SDK sample out!

I made a few small modifications to my code yesterday, including the removal of the Thread.Sleep and now I could go up to 500 kSPS without any major issues, and I had a 8-hour long recording session at 250 kSPS with only ~25 packet losses, which I think is acceptable.

Thank you again for your input!

[attila]

Szia @Andras

I wanted to say: The cAvailable zero can be expected/accepted, specially at lower rates.