Is this how I'm meant to use flash storage in my design?

[pdw]

Hi all,

I’ve been working with a VHDL design that uses block RAM to store some data that I’m processing. I’m able to look up addresses in the BRAM and get values back a cycle later, and it’s all working fine. I now need to store and process more data than can fit into BRAM, so I’m looking at using the on-board flash, which I believe should have more than enough capacity.  For reference I'm currently using a Genesys 2 board, but I'd like my approach to also be valid on an Arty S7 board too.

In Vivado I’ve successfully used flash to store the FPGA bitstream (so that my design survives the board being powered off), and that’s about as much as I’ve ever seen about flash: I’ve no other experience of what to expect when using flash, or AXI, or SPI, etc. But I’ve done some reading around and pieced a few ideas together, and I now think I’ve got a plan for something that might work.

But I’d really appreciate some feedback from people here as to whether I've stumbled upon a reasonable approach, or if I’m missing a much better way to do things.

 

Here’s my current sketch of how I plan to access my data stored on the on-board flash.

• Add an AXI Quad SPI flash IP core to my design, because that’s a thing that knows how to talk to SPI flash.
• Wire up that core into my VHDL logic.
• Work out how to communicate with the flash over AXI (or SPI?), and write some logic for that into my VHDL design.
• Instead of sending the read addresses to BRAM as I currently do, send them instead to this new logic that I’ve written, which will fetch values from the flash storage for me. Be prepared to wait for more cycles than before.
• Instead of incorporating the BRAM data directly in my VHDL files, create a separate data file with the data that I need to store.
• Use the ‘write_cfgmem’ command and the ‘-loaddata’ option to build an MCS file that contains my data alongside the FPGA bitstream.
• Tell Vivado to load that combined result into the flash memory.

Does that (or even any part of that) sound sensible to anyone?  Am I missing a simpler way?

I would ideally like to treat the flash just like a simple ROM (send addresses in, get values out), so if there’s a more direct way to do that I’d really like to be aware of it.  But since I don't know what I don't know, I'd also appreciate any replies that just say "Yes, that's how I'd do things too."

 

Many thanks in advance,

Paul.

[artvvb]

Hi @pdw

1 hour ago, pdw said:

• Add an AXI Quad SPI flash IP core to my design, because that’s a thing that knows how to talk to SPI flash.
• Wire up that core into my VHDL logic.

For your outline as presented, in terms of complexity level, these steps concern me as they'd require building a custom AXI-lite master. If you're dead-set on using AXI Quad-SPI instead of a custom SPI controller, I would first put together a MicroBlaze design that uses it with the xspi drivers, so that you can build an understanding of what the IP's registers are doing and how you need to access them. It's currently undocumented on our reference site, but this demo was put together last year to write and read data from flash on a Nexys Video:

• Demo software main: https://github.com/Digilent/Nexys-Video-SW/blob/0987e797293f60b3caa27a400868bcf51b22c6a4/src/quad_spi/src/Flash_Memory_Test.c
• Demo release page: https://github.com/Digilent/Nexys-Video/releases/tag/QSPI%2F2023.1-1
• A similar demo page which should include the bare bones for opening the project from a release: https://digilent.com/reference/programmable-logic/nexys-video/demos/axi-ps2.
• Generic git documentation: https://digilent.com/reference/programmable-logic/documents/git.

Thanks,

Arthur

[pdw]

Hi @artvvb

19 hours ago, artvvb said:

For your outline as presented, in terms of complexity level, these steps concern me as they'd require building a custom AXI-lite master. If you're dead-set on using AXI Quad-SPI instead of a custom SPI controller, I would first put together a MicroBlaze design

Many thanks for that — this is exactly the sort of feedback I was hoping to get here. Thanks also for pulling together those AXI demo resources (though, as you’ll see, I suspect I won’t be needing them).

I have to say that no, I’m not at all dead-set on using AXI. In fact, I only picked it because I read a comment in the Genesys 2 reference manual that said “The AXI Quad SPI IP core is recommended for easy access to the Flash memory.” But I've never used AXI before, and have no experience with MicroBlaze, or with the graphical IP integrator design tools either. So if there's a simpler way I can make this work, I'll be more than happy to use that instead.

 

From what you wrote, it sounds like the better thing for me to do would be to write some VHDL to talk directly to the flash, over the SPI interface. Am I understanding that right?  Or is there some other approach that I might want to consider?

Thanks again,
Paul.

[artvvb]

Hi Paul,

If it was me, I'd likely try building out a microblaze system first, just because there are more drivers and example sources available. If too many FPGA resources are used, or if the latency involved in the processor receiving a request from hardware, reading data from flash, and forwarding it to the hardware, is too much, then I'd look into building out a custom controller or AXI master. In either case for custom RTL, there would be a fair amount of time spent with either the flash datasheets or AXI QSPI datasheets and simulating the design before going to actual hardware. I'm not sure which would be easier between the AXI design and SPI design - SPI itself is straightforward, but the complexity is in understanding the flash device's command set. I also admittedly have a bias towards processor-based designs.

Thanks,

Arthur

[pdw]

Hi Arthur,

Thanks for those suggestions.  Just to check I've picked up on everything you've said, here's what I'm hearing from you.  (Please correct me if I've misunderstood any of these.)

• Jumping into this by writing my own VHDL to control an AXI Quad SPI flash core would not be a great first step.  (I think "writing VHDL" is what you mean when you talk about producing a "custom controller".)
• If I do end up having to write my own controller, then SPI (i.e. talking directly to the flash) is possibly going to be a simpler target than the AXI QSPI.
• In either case, I'd need to carefully study the flash chip and/or AXI QSPI datasheets, and expect to spend time in simulation to get things right.
   
• A better place to start would be to use a MicroBlaze core (which I think I can program by writing C code?)
• This is easier to explore with, and has more drivers and example resources available for me to draw upon.
• If the result is not too large or too slow, then this MicroBlaze solution might be all that I need.
• My existing VHDL design can send addresses into it, it can retrieve those from flash, and the data can then be passed back into VHDL land.
• Otherwise, if it's not suitable, then the details of the AXI or SPI protocols I've picked up whilst experimenting it will still be useful for building custom controllers anyway.

Is that about right?  In particular, have I understood you correctly that I could potentially use the MicroBlaze to provide flash memory access?

We don't yet know the first thing about processor-based FPGA design here -- is this something to do with the "Vitis" tool that I see in Vivado? -- but it sounds like it will pay off in flexibility once we've gotten into it.  I've found several "Getting Started with MicroBlaze" guides on the Digilent website that I'll try out.

Do you have any other basic "MicroBlaze for Complete Beginners" guides you can recommend?

 

Apologies for packing so many questions into this reply, and thanks again for your help,

Paul.

P.S. In your earlier post, how did you make '@pdw' be displayed as a button?

[artvvb]

Hi Paul,

32 minutes ago, pdw said:

Is that about right?  In particular, have I understood you correctly that I could potentially use the MicroBlaze to provide flash memory access?

Yes, though to reiterate, I'm hedging about whether writing VHDL might be harder than a MicroBlaze design - if I was doing the design, it's what I'd personally look at first, but I have my own biases.

35 minutes ago, pdw said:

We don't yet know the first thing about processor-based FPGA design here -- is this something to do with the "Vitis" tool that I see in Vivado? -- but it sounds like it will pay off in flexibility once we've gotten into it.  I've found several "Getting Started with MicroBlaze" guides on the Digilent website that I'll try out.

Do you have any other basic "MicroBlaze for Complete Beginners" guides you can recommend?

This guide would be the recommended starting point within stuff Digilent hosts: https://digilent.com/reference/programmable-logic/guides/getting-started-with-ipi. Avoid DDR if possible, which would also mean avoiding some larger drivers or C libraries like stdio. You also need a way of moving data between the processor and the rest of your design. For low speeds, AXI GPIOs work (https://forum.digilent.com/topic/28261-qol-script-for-vivado-block-design-and-ps-pl-communication/) - I think I mentioned earlier, but latency and throughput for a microblaze based system will probably be fairly poor relative to a pure-VHDL-based solution, so the amount of data you need to move matters.

When typing out an @, a dialog pops up with usernames, clicking on the user adds in a button, which sends the person a notification. It's helpful in case the person isn't subscribed to the thread.

 

Thanks,

Arthur

[pdw]

Hi @artvvb

[Aha, that does work fine if I'm typing, just not when I copy/paste.  Thanks.]

Cheers for those references.  I've been working through the '.../getting-started-with-ipi' guide that you linked, to try out a basic MicroBlaze C project.  As you suggested, I followed the steps for boards without DDR memory (even though it's present on my board).

I've nearly made it to the end of the guide, but am now hitting this error in Vitis when I try to build the C code:

[...]
/home/pdw/Xilinx/Vitis/2022.1/gnu/microblaze/lin/x86_64-oesdk-linux/usr/bin/microblaze-xilinx-elf/microblaze-xilinx-elf-ld.real: microblaze_flash_wrapper_app.elf section `.heap' will not fit in region `microblaze_0_local_memory_ilmb_bram_if_cntlr_Mem_microblaze_0_local_memory_dlmb_bram_if_cntlr_Mem'
/home/pdw/Xilinx/Vitis/2022.1/gnu/microblaze/lin/x86_64-oesdk-linux/usr/bin/microblaze-xilinx-elf/microblaze-xilinx-elf-ld.real: region `microblaze_0_local_memory_ilmb_bram_if_cntlr_Mem_microblaze_0_local_memory_dlmb_bram_if_cntlr_Mem' overflowed by 1856 bytes
collect2.real: error: ld returned 1 exit status
make: *** [makefile:38: microblaze_flash_wrapper_app.elf] Error 1

My guess is that this error is caused by a mistake I made, much earlier in the tutorial.  I misunderstood what the 'Local Memory' setting was referring to when configuring the MicroBlaze IP block, and so (thinking that I wouldn't need much of it) I set it to 8k rather than 32k.

I've had a poke around in the block diagram and I now can't see any way to alter this setting.  (Specifically, I'm wanting to alter the 'Local Memory' value that was originally set via the Run Block Automation dialog when I added the MicroBlaze IP to the diagram.)  But this isn't too surprising, as I also note that the tutorial, in the section on configuring the MicroBlaze, has this to say:

Quote

while it is possible to change these settings manually later [...], the easiest way to do so will be to clear the Microblaze processor out of your block design and restart the process of adding the processor.

So, I've got two questions at the moment:

1. Could that 'Local Memory' setting really be the cause of the compiler error I'm seeing?
2. If so, is there a better way to fix that setting now, without needing to wipe the block diagram (and rebuild it correctly this time)?

Many thanks,
Paul.

[artvvb]

Hi Paul,

Yeah, this is a result of the local memory being too small. It's come up in other threads, and as I understand it, you can adjust the memory size from the Address Editor:

Thanks,

Arthur

[pdw]

Hi Arthur,

I'm glad I asked — that approach was much easier than rebuilding the block diagram!  Thanks.

I used those steps to increase both the DLMB and the ILMB values in the block design, reverting them from 8K back up to 32K. (In my diagram, unlike the screenshot in the link, they weren't in the same part of the tree: one was in Network 0 and the other in Network 1.  I've no idea if that's important or not, so I presumed it wasn't.) I rebuilt the bitstream and exported the XSA file, and then found and used this guide to update the hardware specification back in Vitis:

https://digilent.com/reference/programmable-logic/guides/vitis-update-hardware-specification

In Vitis, I could then see in the Explorer that the file at ‘<my_project>_wrapper/hw/<my_project>_wrapper.mmi’ had been updated, and that the ‘AddressSpace’ element there now had an ‘End’ attribute set to 32767, as expected.

 

However, the build still doesn’t work. I’ve tried running a ‘Clean’ on each of the platform, system, and application projects in the Assistant area just in case, but I still get this:

17:00:20 **** Incremental Build of configuration Debug for project microblaze_flash_wrapper_app ****
make all
Building target: microblaze_flash_wrapper_app.elf
Invoking: MicroBlaze gcc linker
mb-gcc -Wl,-T -Wl,../src/lscript.ld -L/home/pdw/Documents/Vitis-workspace/microblaze_flash_wrapper/export/microblaze_flash_wrapper/sw/microblaze_flash_wrapper/standalone_microblaze_0/bsplib/lib -mlittle-endian -mcpu=v11.0 -mxl-soft-mul -Wl,--no-relax -Wl,--gc-sections -o "microblaze_flash_wrapper_app.elf"  ./src/main.o   -Wl,--start-group,-lxil,-lgcc,-lc,--end-group
/home/pdw/Xilinx/Vitis/2022.1/gnu/microblaze/lin/x86_64-oesdk-linux/usr/bin/microblaze-xilinx-elf/microblaze-xilinx-elf-ld.real: microblaze_flash_wrapper_app.elf section `.heap' will not fit in region `microblaze_0_local_memory_ilmb_bram_if_cntlr_Mem_microblaze_0_local_memory_dlmb_bram_if_cntlr_Mem'
/home/pdw/Xilinx/Vitis/2022.1/gnu/microblaze/lin/x86_64-oesdk-linux/usr/bin/microblaze-xilinx-elf/microblaze-xilinx-elf-ld.real: region `microblaze_0_local_memory_ilmb_bram_if_cntlr_Mem_microblaze_0_local_memory_dlmb_bram_if_cntlr_Mem' overflowed by 1856 bytes
collect2.real: error: ld returned 1 exit status
make: *** [makefile:38: microblaze_flash_wrapper_app.elf] Error 1

17:00:20 Build Finished (took 207ms)

which as far as I can see, is exactly the same error message as before.

 

Any suggestions of what I might be missing? I'm still at the "read instruction, click button" stage with Vitis, so I'm expecting it's a beginner's mistake somewhere.

Also, in case it’s relevant (but definitely not presuming anything here!), I’m happy to upload the Vitis project somewhere if that’ll help: there’s nothing sensitive in this project yet, and it compresses pretty well.

 

Thanks in advance for any clues you might have,

Cheers,

Paul.

 

P.S. No idea why my previous post was double-posted; is there some way to delete one of them?

[artvvb]

It's odd that the heap is overflowing the local memory by the same number of bytes. Feels like it might indicate that it's still out of sync with the updated spec somehow, but I'm not sure - the instructions you found for switching out the XSA are the correct ones. I'd be curious whether a new application project with the same source files and the new XSA still presents the same bug.

You could also potentially try reducing the heap size in the linker script. It doesn't solve the problem of the updated address map not getting applied, and could lead to overflows, but to just get something working initially... The screenshot below is from a Zynq project with DDR, so yours will look a little different, but this is where to find the heap size setting:

If you end up uploading the Vitis project, please use the File -> Export menu instead of zipping the folder in Explorer, as there are a bunch of absolute paths in various files that the software needs to update when the workspace gets moved or copied.

 

[artvvb]

2 hours ago, pdw said:

 

P.S. No idea why my previous post was double-posted; is there some way to delete one of them?

The ... button on each comment should have the option to delete:

[pdw]

Hi Arthur,

Thanks for pointing me at the ‘…’ button on the duplicate post; it was enough to convince me to try the ‘Hide’ option, which seems to have done the trick. Also thanks for that tip about using ‘File > Export’. I'd not have spotted that myself, and I suspect it will come in very handy under the right circumstances.

 

So: recreating a new application project with the updated (32K) XSA file as you suggested did do the trick, and it now compiles the example main.c code without any problem. Almost everything in the tutorial now works, and I can load the bitstream and executable program onto the FPGA board from Vitis, push its buttons, and light up its LEDs!

To see what had changed, I compared the linker settings between the new project (created afresh from my 32K XSA export) and the old project (created from 8K XSA, then updated to use 32K). Surprisingly, the stack and heap sizes turned out to be the same in both, but there was a difference in the ‘Available Memory Regions’ section. This still showed the old size of 0x1FB0 (~8K) in the old project, rather than 0x7FB0 (~32K) in the new project. I couldn’t find where I'd go to fix that (and no longer needed to), but hopefully others might find this info useful.

(Also for anyone following along who, like me, wasn’t familiar with that Linker Script settings window in the screenshot: it’s not a settings dialog, it's a project file. Just open Explorer > [project] > src > lscript.ld, and check that you’re viewing the ‘Summary’ tab at the bottom of the panel.)

 

Right now, I'm stuck on the printf/serial output, though. In the final step of the tutorial it says

Quote

You will need to click back over to the Vitis Serial Terminal from the Console tab.

My console tab doesn’t appear to have a serial terminal: how do I enable that? None of its icons had any obvious options for it.  (I'm running Vitis IDE v2022.1.0 on Ubuntu 20.04 if that helps.)

And, in anticipation of the next step: on my Genesys 2 board, should I expect this serial output to come back along the JTAG cable, or will I need to connect another cable to the separate ‘UART’ USB socket?

Thanks again for your guidance on this,
Paul.

[artvvb]

Good to hear you're making progress!

5 minutes ago, pdw said:

My console tab doesn’t appear to have a serial terminal: how do I enable that? None of its icons had any obvious options for it.  (I'm running Vitis IDE v2022.1.0 on Ubuntu 20.04 if that helps.)

I'm on Windows and typically use Tera Term or PuTTY for talking to COM ports, rather than the built-in serial terminal. Either PuTTY or something like Minicom might be appropriate.

8 minutes ago, pdw said:

And, in anticipation of the next step: on my Genesys 2 board, should I expect this serial output to come back along the JTAG cable, or will I need to connect another cable to the separate ‘UART’ USB socket?

Yes, you will need to use both ports. This section of the manual describes the USB UART circuitry: https://digilent.com/reference/programmable-logic/genesys-2/reference-manual#usb_uart_bridge_serial_port.

Thanks,

Arthur