Advice on best product for synchronous data acquisition using GNSS timepulses

[DGC]

Hi!

I am currently looking into possible products for data acquisition that will help me conduct some research in remote environments and I am hoping someone might be able to point me in the right direction for the most suitable Digilent product. The application is as follows:

I have multiple "stations" that are spaced over 100km apart. Each "station" will have a single channel data acquisition device that collects precisely timestamped samples. It is important that the samples are synchronously triggered so the samples can be correlated later on. To do this, I have a GNSS receiver that outputs two separate accurate temperature compensated timepulses. One of these timepulses operates at 1MHz, and paces ADC conversions. Another timepulse operates at 1KHz (or there abouts) and paces the filling up of sample buffers. Everytime the 1KHz timepulse is triggered, a timemark is sent from the GNSS receiver to the computer. And once the sample buffer is sent to the computer, this timemark is associated with the sample buffer. Ideally I would like to be able to constantly acquire samples using this method, but I might be able to settle for short breaks in between filling up the sample buffer.

Two Digilent products come to mind.

The first is the MCC USB-2020 operated in BURSTIO mode. However I am not sure if I can continuously sample using this product, as I am not sure if new samples can be stored in the onboard memory while old samples are being sent out via USB. Can someone confirm this behaviour?

The second is the Analog Discovery 3, which would be my preferred method. However I am not sure with the ADC conversion can be paced externally. I figure I can use an External Trigger to pace the filling up of buffers, but can someone confirm that you cannot pace ADC conversions externally? 

I have attached some simple diagrams for these potential setups below. I am also open to other devices and methods, provided that the method will provide a precise level of synchronisation that will allow the sampled data to be correlated successfully.

Thanks in advance!

 

[Guest]

Hi @DGC

What amount of jitter are you willing to accept on the ADC conversions?

The Analog Discovery can be configured to use an external sample clock, but that itself is sampled by the device on its internal 100 (or 125) MHz crystal clock. And this clock is generated by its onboard crystal, and will float a bit relative to your GNSS. This implies that worst case, you could see ~10 ns sample-to-sample jitter. (I don't know the MCC devices well, but I would expect something similar is true for them as well). If that's unacceptable, you will need a different solution altogether (clocking an ADC directly with the 1 MHz sample, for example).

 

With the AD3, it is fully possible to do full gapless recording at 1 MS/s. (That's already possible with the AD2).

I am not entirely sure I understand what your 1 kHz signal is for. I suspect the intent there is to be able to get absolute timestamps of your ADC samples, to be able to correllate the samples recorded at the different sites, right? The reason I ask is that there may be alternative (more robust) ways of achieving that.

For example, an alternative approach would be to record a low-frequency timing signal (such as the GPS'es PPS signal) on the second analog-in input of the recorder, and then locally analyze that channel to write a "timestamp" log of your sample data along with the data itself, for example, on every rising edge of the PPS signal. This will allow you to reconstruct absolute timing and timestamp each individual 1 MHz sample in post-processing. It will also provide you with a highly robust verification that everything works / worked as intended, without glitches.

NB a good thing to check prior to your field test is the stability/reproducibility of the phase offsets of your 1 MHz / 1 kHz signals generated by your 2 separate GNSS receivers. I have seen similar devices that had essentially a random phase (w.r.t. the PPS signal) after each power cycle, which at the very least is important to be aware of. 

 

[DGC]

8 minutes ago, reddish said:

Hi @DGC

What amount of jitter are you willing to accept on the ADC conversions?

The Analog Discovery can be configured to use an external sample clock, but that itself is sampled by the device on its internal 100 (or 125) MHz crystal clock. And this clock is generated by its onboard crystal, and will float a bit relative to your GNSS. This implies that worst case, you could see ~10 ns sample-to-sample jitter. (I don't know the MCC devices well, but I would expect something similar is true for them as well). If that's unacceptable, you will need a different solution altogether (clocking an ADC directly with the 1 MHz sample, for example).

 

With the AD3, it is fully possible to do full gapless recording at 1 MS/s. (That's already possible with the AD2).

I am not entirely sure I understand what your 1 kHz signal is for. I suspect the intent there is to be able to get absolute timestamps of your ADC samples, to be able to correllate the samples recorded at the different sites, right? The reason I ask is that there may be alternative (more robust) ways of achieving that.

For example, an alternative approach would be to record a low-frequency timing signal (such as the GPS'es PPS signal) on the second analog-in input of the recorder, and then locally analyze that channel to write a "timestamp" log of your sample data along with the data itself, for example, on every rising edge of the PPS signal. This will allow you to reconstruct absolute timing and timestamp each individual 1 MHz sample in post-processing. It will also provide you with a highly robust verification that everything works / worked as intended, without glitches.

NB a good thing to check prior to your field test is the stability/reproducibility of the phase offsets of your 1 MHz / 1 kHz signals generated by your 2 separate GNSS receivers. I have seen similar devices that had essentially a random phase (w.r.t. the PPS signal) after each power cycle, which at the very least is important to be aware of. 

 

Hi @reddish

This is actually exactly what I was hoping! Could you tell me how you can configure the use of an external sample clock? I tried finding information like that in the Analog Discovery 3 reference but couldn't find it. I assume it is in the options when using Waveforms? And I assume it is possible to configure this when using the Waveforms SDK also? ~10ns is acceptable for this particular application.

What you said about the 1KHz signal is correct. The idea of the 1KHz signal was to use it as an External Trigger on the oscilloscope to do timestamping. As in, every 1000 samples (from the 1MHz sampled input) would have 1 timestamp. The 1KHz value was chosen arbitrarily, but also so the Analog Discovery 3's buffer would not be completely full. I could also use something slower and probably will once I get my hands on a device and test everything. You idea also would work well! But as far as I understand it, it would double my data throughput by necessitating the sampling of 2 channels at 1MHz instead of one? This would probably be okay for my application, and this solution does seem easier to implement, so I may go down this path.

And thanks for the heads up regarding GNSS phase. The receivers I will test with are specifically designed for timing applications so I certainly hope this doesn't happen... But I will do the testing for sure. 

Thanks for your input, it is much appreciated.

[Guest]

Hi @DGC

The data rate between the AD3 and the PC would indeed double, but the data rate of your data written to disk wouldn't necessarily -- you can do a tiny bit of processing to extract the slow pulse (eg PPS) and write that separately with negligible data rate compared to the 1 MS/s.

I do not know about your application, so I'm not sure if you want to record long stretches of time. It is quite possible to store samples as 16-bit integers rather than floats for a significant saving, without any loss of precision (the API allows for that).

To select a sample clock source, look at the FDwfAnalogInSamplingSource API function.

If it was me I would implement such a data capture program in Python using pydwf - it is by far the easiest way to get something like this up and running (but of course I'm biased because I made pydwf). I'd also recommend writing your captured data in HDF5 using the h5py package, which would allow for data and metadata to live happily next to each other in one file. I don't know how you intend to post-process your data, but essentially all languages that you'd want to do processing in can handle HDF5.

If you want I could cobble together a program (close to) how I would do this, using pydwf and h5py, if only for inspiration.

 

[zygot]

8 hours ago, reddish said:

The Analog Discovery can be configured to use an external sample clock, but that itself is sampled by the device on its internal 100 (or 125) MHz crystal clock. And this clock is generated by its onboard crystal, and will float a bit relative to your GNSS. This implies that worst case, you could see ~10 ns sample-to-sample jitter.

10 ns sample-sample jitter seems a bit optimistic to me if you are asynchronously sampling one of the trigger input pins for synchronization between 2 remote data collection stations. Perhaps +/- 10 ns between stations worst case, depending on other factors? I suspect that the analysis is more complicated than considering the FPGA clocking. The original AD and the AD2 provided a fairly informative description of the circuitry involved but the AD3 is more secretive. It should be noted that none of the Analog Discovery products has a specification for time-base accuracy. The time base design for all of the Analog Discovery products is pretty good, though the clock stability for the 20 MHz DSC1101 isn't specified, so I'm not sure why this isn't specified.

[DGC]

22 hours ago, reddish said:

Hi @DGC

The data rate between the AD3 and the PC would indeed double, but the data rate of your data written to disk wouldn't necessarily -- you can do a tiny bit of processing to extract the slow pulse (eg PPS) and write that separately with negligible data rate compared to the 1 MS/s.

Great Idea. I will definitely look into this.

22 hours ago, reddish said:

I do not know about your application, so I'm not sure if you want to record long stretches of time. It is quite possible to store samples as 16-bit integers rather than floats for a significant saving, without any loss of precision (the API allows for that).

I'll basically be recording between 8 to 10 hours a day, and will store the samples in their raw (integer) format for post processing later. For the correlation I don't need voltage values anyway, just the raw sample values.

22 hours ago, reddish said:

If it was me I would implement such a data capture program in Python using pydwf - it is by far the easiest way to get something like this up and running (but of course I'm biased because I made pydwf). I'd also recommend writing your captured data in HDF5 using the h5py package, which would allow for data and metadata to live happily next to each other in one file. I don't know how you intend to post-process your data, but essentially all languages that you'd want to do processing in can handle HDF5.

If you want I could cobble together a program (close to) how I would do this, using pydwf and h5py, if only for inspiration.

I will definitely check these out, thanks! It will be a little while until I get my hands on an AD3 to be able to start implement this myself. If you have the interest feel free to do something simple and send it my way!

[DGC]

15 hours ago, zygot said:

10 ns sample-sample jitter seems a bit optimistic to me if you are asynchronously sampling one of the trigger input pins for synchronization between 2 remote data collection stations. Perhaps +/- 10 ns between stations worst case, depending on other factors? I suspect that the analysis is more complicated than considering the FPGA clocking. The original AD and the AD2 provided a fairly informative description of the circuitry involved but the AD3 is more secretive. It should be noted that none of the Analog Discovery products has a specification for time-base accuracy. The time base design for all of the Analog Discovery products is pretty good, though the clock stability for the 20 MHz DSC1101 isn't specified, so I'm not sure why this isn't specified.

Thanks for your input on this. Without going into detail the application has other sources of error that introduce a minimum time error equivalent of 100ns. a +-10% addition to this seems acceptable to me. I could in theory accept higher jitter amounts but obviously minimising it would be nice.

I wonder if there is a way I can test this synchronisation accuracy? Perhaps by using the pattern generator and triggering it with the same 1MHz timepulse on two separate AD3's and measuring the pattern generator offset with an external oscilloscope with a higher sampling frequency? 

[Guest]

The most obvious way to test this is to get your measurement recording setup up and running locally, up to and including a processing script that correlates signals, then do a long-duration test run with a synthetic high-frequency signal (a 100 kHz sine or something like that) going into both setups. This should provide enough data to verify your correlation processing, and with a bit of work you should also be able to estimate sample-to-sample jitter from that.

[Guest]

Okay, I made an example how I would do this using pydwf and HDF5. It is more code than I initially expected.

It turned out that at 1 MS/sec it was possible to sometimes lose samples, due to interference with the data writing. For that reason I put the HDF5 writing in its own thread, which greatly diminishes the risk of this (haven't seen it since I did that).

Then, more of an insight: I now sample the main data channel and the PPS signal both at 1 MS/s (assuming an external 1 MHz clock). It then becomes important that the phase relation between the PPS and the 1 MHz is well-defined and identical for the two setups (as previously metioned), but also, that the PPS signal is stable as the sample clock has a rising edge. It is important that you check this and if needed, take measures to ensure this (e.g. use a long cable from the PPS to the Analog Discovery to delay the signal). Use a scope to verify. If this assumption breaks down, there is a risk that the two recorders have a single-sample offset, which could go unnoticed quite easily.

The way to make all this reliable is a test with a synthetic signal as I mentioned above.

 

sample_clock_example.py

[zygot]

It certainly would be nice if Digilent added a description of the AD3 clocking circuitry as it did for the earlier two versions. Potential customers could figure out a lot more for themselves. This quote from the reference manual is inadequate: "The system clock can changed to use an external reference clock ranging between 10 MHz (Megahertz (million times per second)) and 50 MHz (Megahertz (million times per second)) that is provided as an input to the Trigger 1 signal pin." It raises more questions than it resolves.

BTW The statement above would seem to exclude using the Trigger 1 input as an external sample clock of 1 MHz.

Edited August 16, 2023 by zygot

[Guest]

Hi @DGC

This thread shines light on the fact that in principle, the AD3 can use an external reference clock. However, I don't currently know if/how this functionality can be accessed from the public API.

If it would be possible to access this functionality programmatically, an easier way to accomplish what you want would be to use your GNSS as a clock source (assuming it can output a standard 10 MHz clock reference signal), then trigger acquisition on a PPS pulse.

You would then no longer need an external sample clock.

What type of GNSS are you using? I have a super-nice SRS FS740 which can do all this, but it is unfortunately expensive. I am not sure if there are lower-cost GNSS modules out there that can generate a 10 MHz signal, that would be quite awesome.

EDIT: I found some options in the ~ 150-200 euro range by googling that can do a GPS-disciplined 10 MHz, but unfortunately most (all?) of them seem to be lacking the PPS output (with a well-defined phase relation to the 10 MHz output) which is needed for triggering.

Edited August 17, 2023 by reddish

[DGC]

On 8/16/2023 at 5:40 PM, reddish said:

The most obvious way to test this is to get your measurement recording setup up and running locally, up to and including a processing script that correlates signals, then do a long-duration test run with a synthetic high-frequency signal (a 100 kHz sine or something like that) going into both setups. This should provide enough data to verify your correlation processing, and with a bit of work you should also be able to estimate sample-to-sample jitter from that.

Great idea.

On 8/16/2023 at 10:22 PM, reddish said:

Okay, I made an example how I would do this using pydwf and HDF5. It is more code than I initially expected.

It turned out that at 1 MS/sec it was possible to sometimes lose samples, due to interference with the data writing. For that reason I put the HDF5 writing in its own thread, which greatly diminishes the risk of this (haven't seen it since I did that).

Then, more of an insight: I now sample the main data channel and the PPS signal both at 1 MS/s (assuming an external 1 MHz clock). It then becomes important that the phase relation between the PPS and the 1 MHz is well-defined and identical for the two setups (as previously metioned), but also, that the PPS signal is stable as the sample clock has a rising edge. It is important that you check this and if needed, take measures to ensure this (e.g. use a long cable from the PPS to the Analog Discovery to delay the signal). Use a scope to verify. If this assumption breaks down, there is a risk that the two recorders have a single-sample offset, which could go unnoticed quite easily.

The way to make all this reliable is a test with a synthetic signal as I mentioned above.

 

sample_clock_example.py 9.27 kB · 5 downloadsapprecaite

This is really great! I figured as much that a kind of double buffer would be required in the end to be reading and writing data at the same time for sustained high throughput. I appreciate the effort you have gone to. 

On 8/17/2023 at 1:02 PM, reddish said:

Hi @DGC

This thread shines light on the fact that in principle, the AD3 can use an external reference clock. However, I don't currently know if/how this functionality can be accessed from the public API.

If it would be possible to access this functionality programmatically, an easier way to accomplish what you want would be to use your GNSS as a clock source (assuming it can output a standard 10 MHz clock reference signal), then trigger acquisition on a PPS pulse.

You would then no longer need an external sample clock.

What type of GNSS are you using? I have a super-nice SRS FS740 which can do all this, but it is unfortunately expensive. I am not sure if there are lower-cost GNSS modules out there that can generate a 10 MHz signal, that would be quite awesome.

EDIT: I found some options in the ~ 150-200 euro range by googling that can do a GPS-disciplined 10 MHz, but unfortunately most (all?) of them seem to be lacking the PPS output (with a well-defined phase relation to the 10 MHz output) which is needed for triggering.

At the moment I am planning to use a u-blox F9T module (probably the SparkFun breakout that you can find here. It is spec'd to output a 25MHz pulse if required, and has two timepulse outputs so I would use one to generate the external clock, and the other to generate the PPS signal. I have read that thread, and I think my angle of attack will be to provide an external clock between 10MHz and 25MHz, and then average samples until I get my desired sample rate. 

[Guest]

Hi @DGC

The F9T is indeed very suitable. In fact I ordered one of these a few days ago for some fun experimentation, following our discussion last week. The fact that it can output two time-pulses is super useful, this is a feature that is, unfortunately, missing from the newer F10T.

Once it's in (it's on back-order) I can verify its time behavior using my fancy FS740. Typically a PPS pulse of a GNSS receiver has an edge-to-edge jitter of a few nanoseconds. I will surely make an Allen deviation plot to verify timing behavior over a wide range of frequencies, if it's useful I can share that with you -- although perhaps, with your requirement that allows for 100ns of jitter at 1 MHz, that's a bit of overkill -- the F9T should far exceed that if it's properly designed, which I expect it to be.

Myself I ordered the ublox EVK-F9T evaluation kit. It breaks out a few more pins of the F9T by default, but on the other hand, it doesn't have the nice SMA connectors that the Sparkfun version has. Potatos, potatoes, I guess.

I agree that probably the best way will be to feed the 10 MHz as an external clock to the AD3 (assuming it takes 10 MHz, I don't see it documented anywhere), and then trigger on, and/or capture the PPS to allow time attribution for each sample.

In the F9T manual I haven't seen a spec on the phase relation between the two time pulse channels, but from the programming interface they document I suspect what you will see is that the slow PPS output will share its rising edge with the fast time-out pulse rising edge. Now the PLL in the AD3 will up-scale the reference frequency to 100 or 125 MHz, which will have an essentially undefined phase relation to the PPS signal. I would recommend checking if the AD3 and the F9T-generated PPS reproducibly end up at the same phase relation over different power-ups of both devices, by generating a high-frequency digital signal on the AD3 and checking if that always looks the same vs the PPS pulse on a scope, over multiple power cycles. If yes, that's at least one less thing to worry about.

 

 

 

[DGC]

On 8/21/2023 at 3:12 PM, reddish said:

I will surely make an Allen deviation plot to verify timing behavior over a wide range of frequencies, if it's useful I can share that with you

If it is no trouble on your end I would love to see the results!