Loading Image Onto FPGA

[digility]

Hello, I have an image processing project. I've already implemented this project in software, where I upload an image, apply filters, perform histogram equalization, and enhance the image. Now, I need to process the image using an FPGA. What is the most ideal way to achieve this? I have a Basys3 board, and should I use UART communication protocol to obtain the pixel intensity values of the image? If necessary, I can also use boards like Arty A7-100T in the laboratory.

What I essentially want to learn is how to transfer an image into an FPGA?

@asmi @zygot

[zygot]

The options for transmitting data between an application in an FPGA and a PC are limited mostly by the board design. For the BASYS3, this pretty much means a UART. More expensive platforms might provide a MicroSD card, Ethernet, or higher speed USB interface. There are cheap FPGA boards that aren't designed to be limited rather old and tired PMOD boards. I like the Terasic DE0 Nano and variants. You can add you own high speed interfaces to them. The Cyclone 10 LP dev board has a 1 GbE Ethernet and external memory. All of these are in the same general price range of the Basys3.

A UART at 921600 baud provides roughly 92K characters/s data bandwidth. For an application requiring a lot of data transfer, this might mean pretty slow processing times. You can look at this tutorial for ideas on how to speed up a USB UART interface to 1.2 Mbaud: https://forum.digilent.com/topic/24531-debugging-with-the-ftxxxx-mini-modules/
. This involves buying additional hardware, but can be used with any of Digilent's FPGA boards. You don't need to implement the whole debug design, just understand how the HDL UART interface works and how to control it with a PC software application. The nice thing about this approach is that you can transfer binary data easily which requires fewer bytes than 7-bit ascii, which is what most PC serial UART applications want to use.

Another possibility that can be used with any Xilinx FPGA is the JTAG BSCAN capability. This project is a good lesson in using that method: https://forum.digilent.com/topic/17096-busbridge3-high-speed-ftdifpga-interface/

Sometimes people think that they need a big FPGA device with lots of resources to prove a concept. For the most part, this isn't true. A small device might just require doing the same task differently, piecemeal, or slower. Or you can just process smaller datasets. You don't need to spend a lot of money demonstrating that a concept works; unless you are trying to demonstrate the fastest way to do something.. perhaps.

Not having external memory to hold large datasets might be inconvenient, and certainly will limit how fast you might be able to process the data, image processing in particular. That doesn't mean that you can't prove a concept. It might mean demonstrating a logic application that is different than one that would be used in a real world application. Edited November 19, 2023 by zygot

[digility]

Firstly, I apologize for the delayed response. In the provided link, it mentions that the BasyS3's UART speed can go up to 12Mbaud without using any external module:

Even if it's as low as 1Mbaud, it should be more than sufficient for my needs as I'll be processing a single image. My image is 512x640 16 bit and no color information(no rgb), roughly 5 million bits. With a 1Mbaud UART, I can transfer the image into the FPGA's memory in about 5 seconds. I consulted with my professor, and he confirmed that UART would work for this project. What are your thoughts?

 

My question iş how to automatically send all pixel values of the image to the FPGA via the terminal. Manually entering each value is not feasible. I've seen commands for this in Matlab and Python. I'm sure you've used Matlab or Python. Do you have knowledge about this? I'm considering storing 512x640 pixels in the FPGA's memory.

 

@zygot @asmi

[zygot]

As I mention above, the problem with COM or TTY devices is that operating systems want to treat them as 7-bit asii. You can get around this by using 2 ascii characters to create one 8-bit byte. This halves your data rate. It's also terribly inefficient. You have to include logic in your FPGA to do this in your FPGA application. Nonetheless, if you can wait a while to upload image data, this might be your best course of action. [edit] And yes, I've used Python with pyserial to make the software part of this easier.

It's possible to use FTDI devices with the D2XX driver instead of the default VCOM driver, but according to their documentation you must change the configuration EEPROM to use the D2XX driver. There's no switching back and forth in just software. With the D2XX, you have to write you own PC software application; but you can use binary data instead of ascii.

I believe that the two project links that I posted above should provide good information about some other options. Edited December 4, 2023 by zygot

[D@n]

@digility,

When I first started working with FPGAs, my first project was a serial port to Wishbone bus translator.  I call it a "debugging bus".  Using this abstraction, I could then create and send bus commands to the FPGA from a C++ program.  The interface involved has been so robust, that I've now used it on (just about) every project I've worked on since.  Were I to attempt to load an image onto an FPGA, this would be how I would do it.  A couple of C++ programs would adjust the image as necessary from whatever format it came in on, to whatever format I'd need, and off I'd go.

@zygot does present some good architectural questions to consider.  I've (personally) stuck with 7-bit ASCII.  Over time, I've started using the 8th bit (the MSB) to multiplex two streams together (a CPU console, getting mux'd with the debugging bus), so full 8-bit ASCII is quite possible.  The maximum data rate achievable then depends upon the protocol.  One protocol I built and demonstrated in my blog takes 11 hexadecimal characters (8 for a 32b word, 1 control character, a carriage return and a newline) to send 32bits of information.  (29% efficiency.)  I'm more commonly using 6 characters for this purpose (53%).  More recently, I've been testing an approach that will send 32bits of information in only 5 characters (64% efficiency), while still maintaining the ability to use the serial port as a CPU console.  On one project, I've even gone so far as to run this protocol over a network--so the FPGA can go underwater.  In all cases, I've continued to maintain (nearly) the same C++ interface description as I started with.

For the record, while the FTDI chip declares it can send data at 12Mbaud, I've only been successful at 4MBaud and only reliable at 1 MBaud across multiple circuit boards.  I wouldn't set my expectations any higher than that.  (The GbE interface, however, goes *REALLY* fast ... but that's another story, and takes a lot of work to get going--since you'd need to typically support ARP and ICMP protocols, before even touching the UDP/IP required for something like this.  That's just a lot of work for a student project.)

The issue I would worry about, however, is ... where are you going to store this image?  The Basys3 device doesn't have a lot of memory, and you are likely to need that memory for other things as well.  I suppose you might just feed the image directly into your algorithm.  (I've done that too ...)  But either way, you should think through what you want to do and any consequences associated with it.  Is it possible that the serial interface will go too fast?  Do you need handshaking?  All things to consider.

Dan

[zygot]

@Dan,
You too can get 12 Mbaud out of some of the FTDI UART bridge devices like the FT2232H. It's all explained in the "debugging with FTxxxx Mini-Modules" link that I posted above. You need to use the D2XX driver and D2XX API though. On some FPGA boards, like the Mimas-A7 you don't even have to buy any extra hardware. At 12 Mbaud, I suggest using hardware flow control. This is actually a positive thing for most applications. With a 4-channel bridge you can do 4 channels of 12 Mbaud UART transfers, though this requires 16 IO pins to support individual channel hardware control. Don't expect that you can bond four channels of 12 Mbaud UARTs into one 48 Mbaud UART though.

Also, for the record, you don't need to use any Ethernet packet types to transfer data to an FPGA target from a PC. Just abstract away the Ethernet protocol layer by using another FPGA board with a good USB 3.0 interface and an Ethernet PHY. I've done this using an XEM7320 and a SYZYGY ETH PHY to stream packets ( data plus preamble ) between a Genesys2 and a PC software application. This isn't completely full duplex as packets are streamed out of the PC through the USB interface directly to the target FPGA board and simultaneously storing the return packets into DDR, where it can be retrieved later. The trip across the Ethernet cable is full duplex, though the overall application is not. Of course, one can make Ethernet as complicated as one would like if "virtual" Ethernet doesn't float your boat ( or you need all of the software layers for some reason ). 1 channel of 1 GbE Ethernet is "really fast" unless you are transferring 14-bit ADC/DAC samples at full duplex at 100+ Samples/s. But you can easily send transfer 2 25 Ms/ DAC and 2 25 Ms/s ADC data both ways simultaneously through 1 1 GbE Ethernet PHY. I've done it, it isn't hypothetical.

Where there's a need, there's often a solution.

I also use 7-bit ascii to transfer 4-bit data vectors. This is a nice debugging tool as I can just a common serial program like Putty to read or write an arbitrary number of n-bit registers in my HDL application just by typing in a simple command. This Ip usually ends up in all of my new designs for debugging or profiling purposes. Edited December 5, 2023 by zygot

[asmi]

If you need speed, FT60x is a very good option. On my custom board with FT601 I was able to reach about 2.5 Gbps of raw bandwidth (after all overheads are accounted for) over USB 3, and it's interface to FPGA is very simple (though it consumes a lot of IO pins). As far as I remember, FTDI also sells FMC daughterboards with those devices if you don't want to roll your own for some reason.

[zygot]

With regard to the FT60x bridges, I've been using their FMC eval modules for many years. This past year I've gotten back to using them in new projects and discovered that the current D3XX driver performance is considerably slower than the version that was used when I did those old projects. Throughput is about half of what I was getting before. Even using the old bitstreams and rebuilding the software sources using the current driver doesn't help. Raw bandwidth is a nice metric, but not necessarily useful for predicting final software application performance. I've lost most of my interest in D3XX as a result. The Cypress ( NXP?) FX3 is a better bridge device. Opal Kelly's implementation of it is actually pretty good and reliable.

[zygot]

1 GbE is quite nice. USB 3.0 can be 2-3x faster, with a few caveats, perhaps. PCIe can be a few orders of magnitude faster than the best USB 3.0 bridge, though it comes with a few warts.

None of this is germane to the OP who wants to process 512x640x16-bit images that have to be transported on and off the FPGA ( an A35T ) on his BASYS3.

One image is 655360 bytes; considerably more than 512 KB. You aren't going to create a single memory structure in an A35T that large. To make matters worse, the VGA display only supports 12-bit pixel depth.

So, does this mean that the project goals are impossible? I don't know. If the image processing can be done on smaller sections of the image file in succession, then there's hope. I'm pretty sure that most of the time spent on doing this will be moving data onto and off of the FPGA board. That's not necessarily a show-stopper. Having to implement functionality with fewer resources than you need is the SOP of many an engineer. Edited December 6, 2023 by zygot

[D@n]

6 minutes ago, zygot said:

Having to implement functionality with fewer resources than you need is the SOP of many an engineer.

+1 😁