FPGA board selection

[Agustinus]

Hi, 

I am looking for an FPGA platform for a large sensor array system. I would like to hear your recommendation for an optimal evaluation board, and probably with a compatible extension board, that can accomodate 128 sensors working at sampling frequency of 96kHz. I wish you would be happy to provide me some options. I look forward to hearing from you. Thank you in advance.

[zzzhhh]

I think a low-end Arty A7-100T is more than enough. A book "Architecting High-Performance Embedded Systems: Design and build high-performance real-time digital systems based on FPGAs and custom circuits" written by Jim Ledin uses this board to implement a digital oscilloscope sampled at 100 MHz with 14 bits of resolution. Digilent is offering 15% discount right now.

Note however that the step-by-step tutorial of baseline Vivado project (the same as this Digilent tutorial) in this book does not work under current version of Xilinx Vivado/Vitis software (2022.2.2). I still don't know how to use the Ethernet of this board, and you can get no technical support in this regard from Digilent or Xilinx.

If you somehow get through this tutorial, I would greatly appreciate it if you can share with us how you make it.
 

Edited April 24, 2023 by zzzhhh

[asmi]

Before you select a board, you've got to figure out four things:

1. Data source - How are you going to do the capture? Is it going to be a crap ton of ADCs, or some fast multichannel ADCs, or maybe something else?

2. Data input - Whatever you settle on in (1), how this data is going to be fed into FPGA? SPI, I2S, or some custom bus? This will help you determine which and how many pins will you require from FPGA, which can disqualify a lot of FPGA boards because they won't have suitable connectivity.

3. Data processing - Once the question if bringing data into FPGA is resolved, what exactly are you going to do with that data inside FPGA? That has a major implications on a FPGA family and density that you require.

4. Data output - And finally, once you've done all the processing inside FPGA, what are going to be your outputs? This again can disqualify some boards because they lack the output method of your choice.

 

Or, and number 0 - don't ever listen to anyone saying "board X is going to be more than enough" without knowing answers to above four questions, because it will very likely end up with wasted money and frustration.

[zzzhhh]

Except Digilent staffs, some people do not reply in order to answer a question. They only come here to show off. Even after finishing the reply, they still do not know what the question is. Their reply can do nothing but confuse the asker by complicating simple things, with a mild threat. I hope the asker can be smart enough to identify such a people.

[zygot]

10 hours ago, Agustinus said:

recommendation for an optimal evaluation board, and probably with a compatible extension board, that can accomodate 128 sensors working at sampling frequency of 96kHz.

Could you be more specific? What kind(s) of sensors are you talking about? It's hard to provide a good answer without enough information.

In general Digilent FPGA boards are designed to support their PMOD ecosystem, and are IO pin deficient. They do sell a couple of boards with 1 FMC connector but finding an off-the-shelf mezzanine card that supports 128 undefined sensors is going to be a problem. Even if you can list all of the exact sensors by part number an answer is not likely to be a simple one.

Usually, for low Fs, multiple channel ADC applications, analog inputs to the sampler(s) are time mutiplexed. Are you prepared to do some digital and analog design and prototype construction?

[asmi]

3 hours ago, zzzhhh said:

Except Digilent staffs, some people do not reply in order to answer a question. They only come here to show off. Even after finishing the reply, they still do not know what the question is. Their reply can do nothing but confuse the asker by complicating simple things, with a mild threat. I hope the asker can be smart enough to identify such a people.

Well tell us then, how do you know that Arty-100 is going to be "more than enough" without knowing answers to the questions I listed. And, while you're at it, maybe you can enlighten us how that stupid tutorial which you failed to make work is relevant to this topic.

Edited April 24, 2023 by asmi

[JColvin]

Hi @Agustinus,

@asmi and @zygot are correct in that we will need some more details about the types of sensors that you have before any effective answer can be provided. Are these sensors all communicating simultaneously at 96 kHz? Individually addressed like different nodes on an I2C daisy chain that collect data at 96 kHz? Something else entirely?
It would also be very helpful to know the number of individual data inputs and outputs each sensors needs as I would hazard a guess that each sensor is not a one-wire device so you would need much more than 128 spare I/O pins which most of the Digilent and MCC boards (including the Arty A7 100T) do not have available.

Thanks,
JColvin

 

@zzzhhh I am not sure why you brought up the book reference on creating a digital oscilloscope or the Getting Started with Microblaze Servers tutorial, as neither of those things seem to be relevant to the topic at hand of integrating over 100 sensors into a system. I would ask in the interest of clarity and helping the original poster to only address the question at hand. I spoke with @artvvb a bit earlier today and learned that they are still looking into your question on the topic here: https://forum.digilent.com/topic/25583-to-digilent-employees-is-there-any-chance-to-update-the-tutorial-arty-getting-started-with-microblaze-servers/; currently they believe the issue might be due to same changes in DDR clocking, but they are still debugging the issue alongside the myriad of other tasks that they have on their plate as a Digilent employee, though they hope to find a solution in a reasonable time frame.

 

[Agustinus]

@asmi @zygot @JColvin

 

Thank you for your kind responses, appreciated :)

I am sorry for the lack of details. Let me start by answering the questions:

1. Data source - How are you going to do the capture? Is it going to be a crap ton of ADCs, or some fast multichannel ADCs, or maybe something else?

I am planning to use PDM MEMS microphones and group every 8 PDM signals into 1 TDM signal. I am not sure if there is one that can convert 16 PDM signals to 1 TDM signal, maybe I can tristate two TDM signals into one output stream. 

2. Data input - Whatever you settle on in (1), how this data is going to be fed into FPGA? SPI, I2S, or some custom bus? This will help you determine which and how many pins will you require from FPGA, which can disqualify a lot of FPGA boards because they won't have suitable connectivity.

I think I2S interface would be a good for TDM signal. However, the number of I2S interfaces would be limited. If 128 sensor is too many, I guess I can lower the number to 64 sensors. I wish there is an alternative to follow. I am open for any suggestion.

3. Data processing - Once the question if bringing data into FPGA is resolved, what exactly are you going to do with that data inside FPGA? That has a major implications on a FPGA family and density that you require.

One reason to work with the FPGA is to bring large matrix calculations. The input data contains information of amplitude, time, and position. Therefore, it should be put in the shift registers and arranged in a matrix that represents spatial distribution. Then matrix multiplications are performed to fit the data into the model which is representated by multiple transfer function matrices.  

4. Data output - And finally, once you've done all the processing inside FPGA, what are going to be your outputs? This again can disqualify some boards because they lack the output method of your choice.

I like to stream out the output to a MCU or GPU that further processes the output data and presents it as the final output format.

 

Here are my responses for the other comments

In general Digilent FPGA boards are designed to support their PMOD ecosystem, and are IO pin deficient. They do sell a couple of boards with 1 FMC connector but finding an off-the-shelf mezzanine card that supports 128 undefined sensors is going to be a problem. Even if you can list all of the exact sensors by part number an answer is not likely to be a simple one.

You are right, I do not think PMOD ecosystem is sufficient. I am actually considering the FMC connector because I found a commercial product (with xylinx FPGA board) that manages to handle +100 sensors with a couple of FMC connectors. But, I really want to avoid design complexity. 

What do you think about SYZYGY connector?

Usually, for low Fs, multiple channel ADC applications, analog inputs to the sampler(s) are time mutiplexed. Are you prepared to do some digital and analog design and prototype construction?

I guess I could not avoid multiplexing and demultiplexing. I want to groups some sensors to simplify prototype structure; for that reason I choose PDM-based sensor and PDM to TDM signal converters. 

 

@zzzhhh

Thank you for your recommendation. I will check the Arty A7-100T.

 

 

 

 

 

[zygot]

I'm looking at the MP34DT01-M onmi-directional MEMS sensor datasheet. All your FPGA platform would need to do is provide a power supply ( ideally 1.8V ), a L/R select output, a CLK output, and a DATA input for each of the sensors. That's 3 pins per sensor, or 384 pins for 128 sensors, if you were to control 128 separately. Of course, you might not need to do that exactly. If the sensors that you plan on using are similar, then it's an all digital design effort with no SPI or I2C interface... just direct connection. There are complications; there are always complications.

You probably want to spatially arrange the sensors in some sort of configuration, each with a particular orientation in space. If this isn't a 2D configuration, then wiring gets to be a problem.

I'd pay particular attention to how you implement the CLK distribution. A logic derived CLK would be sufficient. You probably don't want to create one CLK output and try and drive all 128 sensors with it, though this would save you 128 pin connections. A tree of external buffers might be good. Separate CLK output pins driving a group of sensors might work as well. It really depends on the physical location and wiring of the sensors. All of those CLK, DATA and L/R lines can be problematic...

The MP34DT01-M can work with 3.3V Vdd, which would be ideal for most FPGA development platforms because most cheap ones only supply a fixed 3.3V Vccio to the IO banks. You need to make sure that the CLK, L/R and DATA signalling is compatible with the IOSTANDARD logic that your FPGA platform can support. For most Digilent FPGA boards this is 3.3V TTL or CMOS. You can always use logic level converters, but this would seriously complicate your prototype. A 3.3V working Vdd introduces its own issues to take into consideration. For this sensor, the L/R has to be either Vdd or GND, not Voh or Vol ( logic high or logic low) This is important to understand.

Even if you can reduce your IO output pin requirements to the minimum this is 128 input pins. You can add digital multiplexers to the design to make the pin requirements a bit more palatable, but at the expense of more complexity and more sampling in-coherency. This is all up to your project requirements.

So far all that I've addressed is the sensor side of your project. What are you planning on doing with the processed data that the FPGA logic produces?

Digilent boards are not ideal for such a project.

I didn't spend a lot of time on this reply, it doesn't represent a robust design concept, and I might not have gotten everything covered correctly, but hopefully provided some things to consider. In the end, you are the one who has to live with you choices.,

Edited April 25, 2023 by zygot

[asmi]

L/R output select is typically hardwired on a board (corresponding to it's position), rather than provided as a variable signal. Also most I2S sensors (including abovementioned MP34DT01-M) are designed to drive L2S data line in pairs (one sensor drives the "left" time slot, another one - the "right" time slot), here is example from the datasheet:

So that by itself cuts down on pins requires to two pins per pair of sensors. Of which the clock line can be shared with all other sensors, using clock buffers and/or delay compensation circuitry if neccessary, because this sensor runs at rather pedestrian 3.25 MHz. So the very minimum for 128 I2S sensors is 64 data pins + 1 clock out. This disquality Arty series boards because they don't have enough pins, but just about any board with FMC connector will easily meet this requirement. You will need to design your own daughterboard for this as I'm not aware of any such board already existing on a market, but that shouldn't really be that much of a problem.

[Agustinus]

@zygot @asmi

 

Thank you for sparing your precious time to provide me some answers. I would like to elaborate my working plan.

About the sensor side, I found a way to group 16 PDM microphones into one TDM/PCM stream. That means, the FPGA platform would be needed to provide 3 pins per group of 16 microphones, resulting a total of 24 pins for 128 microphones. Another thing that would be required to be performed in the FPGA platform is to rearrange/demultiplex each TDM/PCM stream into 16 separate input data. The device that is used to make a group of microphones will be useful to reduce wiring complexity and to help maintaining signal integrity, including the CLK distribution and data buffering.

About the daughterboard, I do not think there is a way to avoid it. I will design a daughterboard and fabricate it.

About the processed data that the FPGA logic produces, I like to stream it out to a GPU or MCU for post-processing and presentation (e.g. save the result to file, show it on the display). I would dedicate FPGA platform only to process raw data because it involves large matrix multiplications.

Frankly speaking, I do not know if Diligent has a matching FPGA development board in their inventory. I would be happy to try one you like to recommend, maybe one that has a FMC connector. I guess it would provide multiple I2S interfaces (8 channels, 24 pins), bit clock at 49.152 MHz (16 slot per-stream, 32 bit per-slot, sampling rate 96kHz), compatible L/R clock (with 50/50 duty cycle), and programming environment that allow direct access to input data/stream and memory/shift-registers. 

I look forward to hearing from you again. Thank you.
 

 

[zygot]

An interface to an external device would involve more FPGA IO pins, unless the platform has a UART or USB interface on-board.  You can always use a couple of pins to use an external USB UART. The Nexys Video and Genesys 2 boards have both UART and a USB 2 interfaces as well as a 1 GbE Ethernet PHY, which can connect the FPGA to a PC. To connect to an uController, SPI ports are likely to be available at decent data rate. SPi is not hard to implement in logic. I2C type interfaces tend to be pretty slow on such devices. There's always Ethernet, though implementing this in logic can be more difficult, though perfectly possible. 10/100 Ethernet is pretty slow compared to something like the Digilent DPTI USB 2.0 interface. It is possible to implement an Ethernet connection entirely in logic ( assuming that your FPGA platform has enough spare pins or an on-board Ethernet PHY ), with no MAC involved and use a software  raw socket implementation on the uControler or PC. How you do this depends on where you want the processed data to end up. There are a number of possible choices. You might even be able to connect your FPGA to a controller using a parallel data interface. I've used the (unofficial) 18-bit SMI interface on RPi 3 and 4 to do this. I've also connected FPGA boards to RPi3, RPi4, and Nvidia platforms over USB using D2XX APi.

However you implement the FPGA/MCU interface in your design I'd recommend avoiding odd clock rates. It's usually easier to use a faster data transport stream interface at an arbitrary data rate and either zero stuff bits or if the data is packetized, adjust the packet payload size. There are a lot of options depending on what the specific devices involved will be. What you choose will largely depend on how complicated you want your logic application and software application(s) to be. It's always a trade-off as far as time and effort goes. As a general rule of thumb any source synchronous interface at around 50 MHz or higher makes using a logic derived clock impractical and you'll need to use FPGA clocking resources.

Lastly, give some thought to elastic buffer memory in your FPGA design to accommodate software latencies in the MCU/GPU implementation of the data stream interface. Most modern FPGA devices like the Artix and Cyclone V will have sufficient BRAM to implement small FIFOs or DPRAM for this. It's easy to prototype a concept in a logic simulation for a specific FPGA variant to help choose your FPGA device and platform. I'd guess that an A50T would be sufficient... only a guessitmate.

Designing and assembling an FMC mezzanine card is not a trivial or inexpensive endeavor if you are only building 1 or 2 prototypes. 

Be aware that most PMOD implementations have 200 ohm series resistors. These can always be shorted or replaced with 0 ohm shunts. Digilent PMOD pin assignments are arbitrary and not designed for wide busses; another reason why they aren't ideal for this kind of project.

Seems, like you are on your way. There are cheap Cyclone V or Artix based FPGA platforms available that should be suitable for a quick prototype construction. HSMC is an alternative to FMC and might be easier to use.

Edited April 26, 2023 by zygot

[asmi]

8 hours ago, Agustinus said:

About the processed data that the FPGA logic produces, I like to stream it out to a GPU or MCU for post-processing and presentation (e.g. save the result to file, show it on the display). I would dedicate FPGA platform only to process raw data because it involves large matrix multiplications.

How exactly are you planning to do that? What protocol/bus/connection? I don't know if your FPGA processing is going to involve any data reduction, because as it is, we're talking about 128 * 96K * 8(bits per sample) ~ almost 100 Mbit/s if using 8 bit samples. So you will need to use 1G Ethernet to stream it out, or PCIE if you prefer internal bus. You can also use USB 2.0, but in my experience USB can be seriously finicky, so I personally try to avoid it unless absolutely neccessary.

Also you might want to design a processing core that you plan to use in FPGA - just assume input data is in some sort of FIFO, and output into another FIFO, and do a functional sims to confirm it works. Once that is done, you can run synthesis to get a rough idea regarding amount of FPGA resources your solution will require. This will help you pick the right density FPGA for the job.

[zygot]

Terasic still sells the kind of FPGA boards that are ideal for these kinds of projects. The DE0 Nano and DE0-CV both have 68 IO pins on 2 40-pin headers. They are relatively cheap. The Cyclone GX Starter Kit has one 40-pin header and 1 HSMC connector with a lot more connectivity options. All of them have easy to use SDRAM. The GX board also has less-than easy to use LPDDR2, but that's only because the free Quartus is really badly supported, as is DDR IP for the cheaper Intel families. But I've done BL8 DDR designs for the GX board after a LOT of effort.

Gee, if only there was an AMD/XILINX based vendor of FPGA development boards that had an interest in this market. Hmmmm.... if only I could think of a company with years of XILINX board design experience.... ideas anyone? Edited April 27, 2023 by zygot

[zygot]

I have no idea why this just now registered in my mind as I was busy doing something completely unrelated... but, about matrix multiplication.

I highly suggest that you prototype your logic design in simulation for a particular board before making any assumptions about what you can do with the hard multipliers in Series 7 devices. Yes the DSP blocks are very fast, ~500 MHz on even the Artix. Except for particular topologies, the fabric connectivity of DSPs is not great. DSP slices have 2 DSP blocks and work well together. When connecting groups of DSP slices into a computational structure timing can get pretty bad. It's something that you will want to research before committing to a design concept.

[edit] You'll need to run implementation to get timing information. You can do a timing simulation on the resulting netlist or just review the reports. For all logic designs with no soft-processor simulation is generally quicker than generating a bitstream if you are adjusting the design and there are no long self-calibration or delay elements in the design. My point is that, with any programmable logic vendor device, NEVER make assumptions about using DSP blocks. Edited April 27, 2023 by zygot

[Agustinus]

@zygot @asmi

Thank you so much for the constructive feedbacks, I surely do some follow-ups. Regarding your comments, yes, the input-output process is following a FIFO rule. I am also considering to reduce the output data rate which is an image stream, i.e., by selecting optimum frame rate and resolution. For the post-processing, I will use Nvidia/Jetson Xavier development board.

I like the idea to start quick prototyping with DE0-CV/Cyclone V development board. I will connect it to a couple of PDM/TDM converter boards. I hope it can help me to be familiar with FPGA ‘programming’. Later, I will move to the ARTIX 7 to simulate large matrix multiplication. Then, after feeling confident, I will develop the daughterboard. I will share you the progress after a while. 

Best regards

 

[zygot]

8 hours ago, Agustinus said:

I like the idea to start quick prototyping with DE0-CV/Cyclone V development board. I will connect it to a couple of PDM/TDM converter boards. I hope it can help me to be familiar with FPGA ‘programming’.

In case this indicates that you are short on programmable logic development experience, I'd suggest choosing a vendor and sticking with it. A significant part of FPGA development is dealing with the tools. In terms of synthesis, Verilog and VHDL are not really programming languages. Each device family from a vendor has specific libraries supporting the architecture and resources of that family. Some aspects of FPGA resources require device specific library support ( for simulation ), like clocking and block memory.These differences will show up, especially from vendor to vendor. The tools, and even the versions of tools have their own quirks, and bugs to deal with. Quartus used to provide a free version of ModelSim for simulation, until version 20 or so, but now that's gone in favor of an Intel simulator. If you have an older installed version of Quartus available, then ModelSim is an options. You can't download anhy Quartus installer now that has it. I'm trying to say that dealing with 2 sets of vendor devices and tools is a big complication for experienced FPGA developers; so much more so for someone "learning on the job".  Even the simulation work flow is different for different vendors tools. I mention this only in case you might develop additional questions.

Edited April 28, 2023 by zygot

[Agustinus]

@zygot

Thank you for your kind suggestion. In deed, I reconsidered my choise after seeing posible complication when changing the vendor. So, I decided to buy the Terasic DE-10 Standard Development Board. I am expecting to stick on it for good. Best regards.

 

[zygot]

9 hours ago, Agustinus said:

So, I decided to buy the Terasic DE-10 Standard Development Board

There are few things to mention with regard to using Intel tools that might not be obvious to everyone.

First of all,.Intel has a loathsome marketing policy designed to extract maximum money out of it's customers. The free version of Quartus is generally fine for MAX10, Cyclone 10LP, and Cyclone IV or V devices, if you don't use external DDR or exotic features like transceivers. You will have to pay handsomely for a tool license for other devices and families. Companies that only design with Intel FPGA devices likely have to pay for 2-3 licenses to cover all of the devices that they might want to use. Just be aware of this.

Every newcomer to FPGA development thinks that an FPGA with an ARM core block in it must be better. It certainly is harder to develop as you need to master both the HW and SW tools. For a product that is micro-controller based ZYNQ or SOC devices may indeed be the way to go. Make sure that you understand the host requirements for the EDS tools before purchasing a board. ( the same can be said for AMD/XILINX VItis if you need to run a Linux OS on your ZYNQ ). I didn't mention any of Terasic's SOC boards because I really, really dislike their ARM toolchain. Developing a complete HW/SW set of applications for a project is much more complicated than just a logic application if none of the vendor IP provides a magic solution. Your application is not one that begs for an embedded processor; it can more easily be implemented in logic. FPGA devices are complicated. FPGA devices with hard processor blocks are 10X as complicated for real world projects. Then, apply another multiplier for learning how to get along with the tools. It's the subtle details that cause the most development effort. If you have experience with nVidia embedded platforms, then you have an idea of what I'm talking about. This can get considerably more difficult if you have to integrate new hardware support into your nvidia-debian kernel using the nvidia software development tools.

The DE-10 may be your best solution, I can't evaluate this. Just consider all of the potential hazards before making a commitment. Getting a heads-up from experienced developers is a good way to expose hazards that are hidden by the advertising glitz. Unfortunately, it's not practical to launch into a comprehensive discourse when replying to a question like yours. I am frequently criticized for being too wordy as it is. For the record, I do not make recommendations about product purchases. I can, however, provide guidance with regard to tools and platforms that I've spent time using and impart a bit of hard earned wisdom (opinions based on experience) along the way. 

I will always advise getting some experience with the tools before spending money on hardware. For devices supported in the free version of the tools you can get a good idea of what you are getting into as well as suitability of a particular device for your project without any expenditure. For programmable devices with hard ARM cores, this becomes a bit more complicated as there's no simulation tool that covers the logic and processor simulation. But you can see what a typical software application development flow looks like. Learning all of this after making a committment and while you are doing your project development is not a happy place.

Edited May 1, 2023 by zygot

[zygot]

If there's ever been a programmable logic vendor with more than 2% market share I've used their tools and devices. Even still, my experience is limited as far as all of the device possibilities and application possibilities are concerned. Sadly, for the kinds of projects that I do, for the case where a general purpose FPGA board can substitute for actual custom designed application specific hardware, Intel based platforms are better than AMD based platforms, in general. This is certainly true for applications requiring lots of IO or differential signalling or when I want to integrate a platform into a solution. It's true for PCIe based applications when everything, including software is taken into account. HSMC prototyping options, if you don't want to build your own boards is better than FMC options. There are exceptions. If you want to experiment with transceivers for communications or video then there are AMD based boards, like the Genesys2,  that provide better options. Frankly,  I prefer AMD/XILINX devices over Intel devices (for the low end free to use devices ). AMD/XILINX tools and device support for low budget development is superior and wider. If you can afford high annual tool expenditures for high-end device support, or have an academic free-pass,  then the calculus might be different. There just aren't any board vendors supplying great platforms at a reasonable price for AMD/XILINX development for the kinds of projects that I want to do. It's a frustrating experience for the experienced FPGA developer and absolutely bewildering for the average newcomer.

Edited May 1, 2023 by zygot

[Agustinus]

@zygot

Thank you for your kind advice. I am a newcomer to FPGA development work. So finding some support materials that help me to learn FPGA quickly is really a matter. If the free version of Quartus is generally fine for Cyclone V device/DE-10 standard development board, then I will take the chance. I can follow some tutorials and examples provided in the bundle. If the ARM core can be utilized to post-process the outcomes of calculation by FPGA, I do not think I need to use NVidia Jetson for doing the same work. But, as you said, I need to expect some complication in the process. I understand the uncertainty. Anyhow, I need to start experiencing FPGA 'programming' work somehow. I will share the progress later. Thank you.

[zygot]

For some endeavors, the distance between two points, such as starting from zero experience and ending up with a completed working design running on hardware, is usually not a straight line. Everyone learns differently and at a different pace. Be prepared for the unexpected and you'll be fine. Mere mortals like me have learned that starting off simple and adding complexity as I gain experience generally works out to be the quicker route to success. But that's what seems to work best for me, most of the time....

[edit] It's always tempting to ask yourself "can X platform be used to accomplish Y goals?" instead of "is there sufficient evidence that I can use X platform to complete Y goals within my budget and time constraints?" What you don't know can hurt you. When evaluating suitability for a particular platform it helps to see what the platform vendor demos for their products offer, and what version of the tools were used to implement them. I tend to be cautious because I know that facts out of context, and advertising promises rarely hold up when I'm doing actual real-world project development. Bord capabilities and board functionality are not the same. There are a lot of "ifs" and "exceptions" lurking in the details that the advertising doesn't cover. Again, I'm me doing "me projects". Your mileage, or download speed, or whatever may vary... Edited May 2, 2023 by zygot