NetFPGA SUME failing acceptance test

[aceuno]

Hi,

We have 2 NetFPGA SUME boards that are failing the tests. 

We followed instructions to set up the environment, using Vivado 2020.1 on Ubuntu 20.04.
We were able to connect to the board through USB using the Vivado hardware server, but it says the debug core is missing.
When running `make test`, we repeatedly see `End of startup status: LOW` and skip all but one of the tests. The one test it does run `CPLD` occasionally passes, but often fails.

Another separate issue is that it seems our system is unable to detect the FPGA through PCIe.
`lspci` command does not have anything related to the board, and there is no difference between two lspci runs, one with the board in the slot (with ld10, power led and ld4, done led both on) and one without the board. We tried to follow some instructions that installed some drivers and it didn’t change anything, but independent from drivers, we believe the device should be picked up by the PCIe slot regardless.

Do you have any recommendations for what to try to fix these issues?
We are also happy to provide additional information for you

Thanks.
 

[zygot]

11 hours ago, aceuno said:

independent from drivers, we believe the device should be picked up by the PCIe slot regardless.

PCIe functionality for FPGA based boards involves logic resources, so the FPGA has to be configured with an application that implements PCIe functionality prior to when a PC detects PCIe devices in your PC. This occurs at the BIOS level before the OS is booted. In order to do that you need to configure the FPGA from the onboard FLASH between the time that the board power supplies are up and stable and before BIOS detects PCIe devices.

How this works depends on how you power the board. If the board is powered by the 12V and 3V supplies from the PCIe slot connector pins and you auto-configure the FPGA from FLASH, then you should be OK. Newer motherboards and power supplies might not support the power requirements for 12V for FPGA boards. The alternative is to power your FPGA board externally so that you can configure the FPGA and not lose configuration when the PCIe slot 12V disappears, ( during reset for instance ). For developing code this is probably the only realistic way to use such a board. This method can cause problems, especially in newer generation motherboards depending on the board power supply design. Usually PCIe equipped FPGA boards have a simple blocking diode to prevent a powered FPGA 12V from driving current into the motherboard.

FPGA boards designed for older generation PCs likely have a number of issues if used in a current generation motherboard.

It's not clear from your question how you are using the board in the context of the quote above but perhaps this reply will help clarify the discussion.

[aceuno]

4 hours ago, zygot said:

PCIe functionality for FPGA based boards involves logic resources, so the FPGA has to be configured with an application that implements PCIe functionality prior to when a PC detects PCIe devices in your PC. This occurs at the BIOS level before the OS is booted. In order to do that you need to configure the FPGA from the onboard FLASH between the time that the board power supplies are up and stable and before BIOS detects PCIe devices.

How this works depends on how you power the board. If the board is powered by the 12V and 3V supplies from the PCIe slot connector pins and you auto-configure the FPGA from FLASH, then you should be OK. Newer motherboards and power supplies might not support the power requirements for 12V for FPGA boards. The alternative is to power your FPGA board externally so that you can configure the FPGA and not lose configuration when the PCIe slot 12V disappears, ( during reset for instance ). For developing code this is probably the only realistic way to use such a board. This method can cause problems, especially in newer generation motherboards depending on the board power supply design. Usually PCIe equipped FPGA boards have a simple blocking diode to prevent a powered FPGA 12V from driving current into the motherboard.

FPGA boards designed for older generation PCs likely have a number of issues if used in a current generation motherboard.

It's not clear from your question how you are using the board in the context of the quote above but perhaps this reply will help clarify the discussion.

Thank you for your reply.

If I'm understanding correctly, the FGPA SUME board has a PCIe power supply connector, and that power supply from the PCIe slot only is not sufficient to power the board.

There are two ways to connect the PCIE power supply:

1. choose to connect it through the motherboard's PCIe power socket or

2. connect it to an external power supply directly, independent from the PC.

To pickup the board as a PCIe device (ie, listed using lspci)

If I use configuration 1, then I need to program the board using the PC, and make the FPGA auto-configure from FLASH, then restart the PC

If I use configuration 2, then I can program the board, and restart the PC without needing to auto-configure.

Is my understanding correct?

Thanks again

Ricky

Edited August 11, 2023 by aceuno

[zygot]

1 hour ago, aceuno said:

FGPA SUME board has a PCIe power supply connector, and that power supply from the PCIe slot only is not sufficient to power the board.

I have no experience with your board specifically. I have used a number of FPGA boards with a PCIe slot edge interface with numerous PCs using a wide range of motherboard/CPU/Power supply generations. What I'm saying is that you can't assume that your PC can power an FPGA board, in particular a Virtex board from the PCIe connector on your motherboard. You need to read the documentation for your NetFPGA SUMI and your motherboard ( at least ) in order to determine if PCIe slot power is sufficient. Newer motherboards might only provide 25W of 12V to the PCIe slot. If you have an expensive GPU card in your PC then there might be other things to consider, such as if you even have a way to power your FPGA card from the PC power supply.

1 hour ago, aceuno said:

There are two ways to connect the PCIE power supply:

You can power your NetFPGA SUMI in 3 possible configurations:

1. From the PCIe slot 12V and 3V pins. Not all motherboards can accommodate this option.
2. From the external power connector on your board and an external power supply.
3. From the external power connector connected to a PC power supply PCIe connector. In this case you MUST verify that the 12V/GND pins are compatible and that your board only requires external 12V supply.

If you are using configuration 1 or 3 then you must setup your board to configure the FPGA from FLASH and hope that configuration takes place prior to when your BIOS detects PCIe devices. If you are using option 2, then you typically power your FPGA board prior to turning it on. You then configure the FPGA. Then you restart your PC so that the BIOS detects a working PCIe device. As long as the external power supply never shuts down, you won't lose FPGA configuration.. which is needed to allow the BIOS to detect it.

I'm talking about general concepts, not providing specific instructions for your board.

One thing to keep in mind when using PCIe for FPGA applications is that you OS likely makes heavy use of the PCIe bandwidth for display purposes.

Another thing to consider is the gap between OS and motherboard/BIOS vendors. I have a i13/Z790 CPU/Chipset motherboard running Ubuntu 22.04. This combination has issues with suspend/resume that occasionally cause the GPU to fail to come out of suspend mode. That's just baseline operation. When I add a PCIe FPGA card things can get crazy.

A lot of very unhappy consequences can arise from debugging PCIe HW/SW applications, such as having to reboot your PC by using the power button. In this case Linux file corruption issues can be a pain to deal with. I prefer using a cheap SBC like the ODYSSEY and an M.2-PCIe adapter cable where really bad days don't mean losing lots of money or years of work.

The biggest problem with cheap PCIe FPGA cards ( including the NetFPGA SUMI ) is the drivers. Good Windows drivers are rarely provided and Linux driver code is rarely up to date with the latest OS distributions or motherboard chipsets. Rarely do they have full PCIe functionality.   

One option is to use Xillybus for PCIe and communicate with your board in device mode. Unfortunately, the free demo versions are limited in performance. The good news is that the PCIe drivers are incorporated in the Linux distribution and are robust ( though I have been unable to build them for Ubuntu 22.04 as yet...)

Edited August 11, 2023 by zygot

[zygot]

I forgot to mention this. Assuming that your FPGA board FPGA device is configured with a working PCIe application you still might have hurdles to overcome.

There are likely BIOS chipset options that by default will not allow your OS to use your board's PCIe interface even if the BIOS detects it. One such setting is the PCIe switch that can be controlled by the chipset or by software. This really depends on how recent your PC hardware is and how up to date your BIOS is. There are a lot of duckies to get in a row before using PCIe with programmable logic.

Intel and AMD don't typically offer very good FPGA PCIe driver support except for their high-end accelerator card portfolio. What would be the business case for doing that?

Edited August 11, 2023 by zygot