Zmod AWG 1411 reset problem

[Todd-Kairos]

Problem summary: I'm using a Zmod AWG 1411 with an Opal Kelly Brain-1 baseboard and experiencing problems with the reset logic on the Zmod AWG. Whenever I load the image via JTAG cable (Digilent JTAG-HS3), it always immediately gives clean sinusoidal outputs as expected. But often when the system boots off an SD card, it comes up in a state where the DAC outputs are scrambled with large discrete jumps instead of the expected sinusoidal outputs. If I manually generate a logical reset by pressing a user switch (more details in the next paragraph), it always recovers to clean sinusoidal outputs after a single reset - whether it was previously in a correct or incorrect state.

Design info: My FPGA code uses a clock wizard with a 200 MHz input to generate 100 MHz (used for throughout my design and for ZmodAWGController:SysClk100 & DAC_InIO_Clk) and 100 MHz/90 (ZmodAWGController:DAC_Clk)) deg phase shifted output clocks - safe clock startup mode is currently off. I have separate Xilinx DDS IP modules driving each of the two DAC channel signals, and a Xilinx Processor System Reset module (includes a clock wizard lock signal input) to reset the DDS and Zmod control IP modules using the peripheral_aresetn output. My project is based off the Opal Kelly Brain "Hello World" example, and consists only of PL code - no PS code. The Zmod controller is the second generation version dated 3/8/2023 (commit #3819e788427d0f70f8d7948c38bcd97b26925157) from the Digilent repository. I mapped Brain user pushbutton #S2 to the auxiliary reset input of the Processor System Reset module. I wanted to rule out that there was any hardware side-effect of the button press (such as loading a power supply), so I wrote some simple logic to generate a reset signal when the switch is depressed and then released.

Reset details: As far as I can tell, the DDS always resets correctly (outputs always look correct using the JTAG/ChipScope ILA). The Zmod controller is supposed to have an asynchronous active low reset that must be held low for >2 clock cycles, which should be satisfied by the Processor System Reset peripheral reset output. I don't know if the reset problems are coming from the Zmod controller FPGA module or from the underlying DAC chip. I've tried a variety of strategies to automatically generate a reset on startup that will act the same as manually pressing the reset button and ensure it always comes up in a correct state, but this has not been fully satisfactory. The most successful strategy so far (though it is highly unsatisfying to rely on an arbitrary empirically determined procedure like this) has been to wait a specified number of clocks after the clock wizard lock goes valid (e.g. 5 sec), then execute a number of periodic resets (e.g. 15x) every second. I trigger resets via a custom RTL module that connects to the Processor System Reset ext_reset_in input, and the peripheral_aresetn connects to the DDS aresetn and ZmodAWGController aRst_n inputs. When a reset takes effect properly, it causes a click sound due to the output enable relay on the AWG board. In some cases asserting a reset does not generate a click sound, so I may only hear 7 clicks out of the series of 15 attempts. Simpler ideas I tried like waiting a period a time and then only issuing 1 or 2 resets gave inconsistent performance. I want to make the system boot up deterministically so I have 100% confidence it will run correctly. It is hard for my FPGA code to tell if everything is OK, though it is easy to see visually from the scope output. I'm currently monitoring ZmodAWGController:sZmodDAC_EnOut and cDataAxisTvalid, and issuing another reset if either is deasserted.

Questions:

1. Why am I seeing different startup behavior when booting off the SD card compared to using the JTAG cable?
2. Supposedly the ZmodAWGController just needs an asynchronous inverted reset for >= 2 cycles. Is there some other important phase that is important, for example, perhaps one should avoid reseting in the middle of Zmod internal SPI commands, etc.? I could potentially check when Output Enable is first asserted as way to synchronize to the internal state if necessary. Or can there be a discrepancy between the ZmodAWGController and the underlying DAC chip?
3. Is there a way to steal a copy of the ZmodAWGController:dZmodDAC_Data[13:0] ODDR output signals without having to modify the ZmodAWGController code? If so, I could check for discontinuities in this data to recognize a bad state.

Thanks very much for any insights!

JColvin edit: confirmed the two google drive links are to .jpg images

[artvvb]

Hi @Todd-Kairos

I've passed your questions on internally to get some more insight on the reset behavior and startup process.

We don't directly support the Brain-1 and haven't tested with it. That said, there shouldn't be any primitive differences between the XC7Z012S part and the Eclypse's XC7Z020 that would cause the described behavior.

For #3, from the synthesized design, you can mark signals inside of the IP for debug with an ILA: https://docs.xilinx.com/r/en-US/ug936-vivado-tutorial-programming-debugging/Adding-Debug-Nets-to-the-Project.

Thanks,

Arthur

[Todd-Kairos]

Hi Arthur,

I tried to follow the tutorial link you sent to find a way to debug the ZmodAWGController:dZmodDAC_Data[13:0] outputs. From the Synthesis Debug view netlist, I can see nets dZmodDAC_Data_0(14) and dZmodDAC_Data_0_OBUF(14). I could not mark dZmodDAC_Data_0 as debug (option is grayed out), but I could and did mark dZmodDAC_Data_0_OBUF as debug. Synthesis works, but Implementation fails and I get error messages to the effect that ODDR loads should only be an output buffer or a port, but it is driving an invalid load. Any other ideas how to handle this? I suppose editing the ZmodAWGController to add a set of non ODDR outputs would work, but I'm not sure if I can just edit the VHDL files or if I would need to make changes using the IP packaging tools (I don't have experience using those yet).

Thanks.

Todd

2 hours ago, artvvb said:

For #3, from the synthesized design, you can mark signals inside of the IP for debug with an ILA: https://docs.xilinx.com/r/en-US/ug936-vivado-tutorial-programming-debugging/Adding-Debug-Nets-to-the-Project.

[artvvb]

Routing the ODDR outputs to other FPGA logic would lead to the same error as marking them for debug. Debugging ODDR inputs might be the better move, or some other signal further back along the signal path. Given that the ILA uses the same clock as the logic it's looking at, there's likely additional complexity around making sure it's out of reset when you want to capture data. It also wouldn't be helpful as much in production as it would be for debugging at the desk.

It ought to be worthwhile in particular to check the data going into the IP. The output waveforms look like what might happen with some inverted bits.

For editing the VHDL sources, using the IP packager would be recommended. There are some checksums and IP version/revision numbers that get updated when you repackage. The packager is also required to add new ports (outside of manually editing generated XML).

Thanks,

Arthur

[Xband]

This seems to have some similarity to the question I just posted.  My problem was the reset in the  "processing_system7_0" for some reason is inverted with respect to the clock wizards signal.  

My error after validation, 

[BD 41-238] Port/Pin property POLARITY does not match between /clk_wiz/reset(ACTIVE_HIGH) and /processing_system7_0/FCLK_RESET0_N(ACTIVE_LOW)
 

AMD issue discusses differences in behavior's between using the boot.ini file and a different JTAG setting;

https://support.xilinx.com/s/question/0D52E00006lLgyWSAS/polarity-of-reset-issue?language=en_US

[artvvb]

Some additional insights from another engineer here that could help for debugging:

On question #2, there aren't any other requirements for the AWG reset input.

On question #1:

• If resets are occurring frequently enough, there may not be time for the output relay to click, if the initialization sequence for the DAC is interrupted. This depends on when sDAC_EnIn is asserted. sInitDoneDAC would tell you whether the initialization sequence is complete.
• Connecting your resets (peripheral_aresetn, ext_reset_in, possibly the PS fclk reset, as XBand mentions above) and clock wizard lock to FPGA pins to forward to an external monitor would let you compare the timing of these signals between JTAG and SD card boot.
• As previously mentioned, extracting the parallel data being sent to the DAC and looking at it on a logic analyzer would be helpful, whether pulling it out of the IP as discussed, or by forwarding it from the AXI stream interface at the IP input.

Thanks,

Arthur

[Todd-Kairos]

Hi Arthur,

Thanks for this information. I haven't been monitoring ZmodAWGController:sInitDoneDAC or sConfigError, but I'm typically waiting 1 sec between reset attempts so I don't think there is a problem of not waiting long enough. I may try to measure the reset behavior following your suggestion above, by adding a breakout board to an unused SYZYGY port and routing copies of interesting reset-related signals to pins I can monitor on an oscilloscope (it is difficult to monitor initial reset behavior on the ILA when booting from the SD card because it takes a while to start/trigger the ILA). I posted a copy of this project just now as it relates to a separate question if you want to take a look. 

Thanks.

Todd