Multiple AXI BRAM and GPIO issues using a ZYNQ PS and RISC-V softcore

[Embargo3198]

Hi, I'm having some problems with implementing a RISC-V SoC using a PicoRV32 soft core.
The board I'm using is PYNQ-Z2, the FPGA part is XC7Z020 and the IDE is Vivado/Vitis 2023.2.2.

The design is visible in the two attached PDFs exported from Vivado, one in default and the other in addressing view. You can also see my current settings in the Address Editor in the picture attached below. I also attached my Vitis program.

The idea is to use a PC to send commands and data to the PS using UART.
For example, after booting the board, you send the command to write the program instructions to memory, start the RISC-V core by writing to an address which controls the enable signal using AXI GPIO and when the core sends a halt signal (also controlled by AXI GPIO) or the PS receives a stop command, you read a segment of memory the core wrote to.

The basic workflow described above is functional, but I still have some problems:

1. AXI peripherals only work if their base addresses are 0x4000_0000, 0x4200_0000, 0x4400_0000 or 0x4600_0000

Anything other than that, be it 0x4100_0000, 0x4001_0000 or 0x4800_0000, just doesn't work and both the PS and RISC-V core won't read or write to those addresses. The PS blocks on reading, which I confirmed when debugging and I'm not sure what the core does.

The PS writing directly to AXI GPIO which controls the onboard LEDs works fine when its address is 0x4600_0000 but not when it's 0x4800_0000.

There are no addressing overlaps or such and the design validation I run every time passes normally.

For some reason, the AXI GPIO registers I'm writing to from the PS's second AXI GP master port through "cpu_ctrl_interconnect" work fine with packed addresses. Although, just swapping 0x4000_0000 for 0x8000_0000 and the PS's AXI ports yields the same result.

 

2. Third AXI memory controller/BRAM generator pair doesn't work

I'm currently using 2 AXI BRAMs, one for the program and data, and the other as a stack. I'd like to add a third one but it seems broken. Writing to it is seemingly fine, but reading just gives a sequence of 0x0000_0000/0xFFFF_FFFF/0x0000_0000 etc.

The same is true even if I remove the AXI GPIOs so it could use one of the working base addresses.
 

3. When using AXI GPIO only the first channel is functional

If I try to use the second channel and configure it, it just doesn't work. As per the manual, the GPIO2_DATA register should be on a +0x8 offset from the base address, but writing to the address does nothing and reading from it gives zeroes. The first channel still works normally, though.

 

I don't know if this is a bug or a normal FPGA part limitation but:
4. The AXI BRAMs are limited to 32 kB
When choosing more than 32 kB, for instance 64 kB, only the first 32 kB (up to base + 0x7FFF) are usable and writing or reading from it doesn't work and blocks the PS indefinitely. I don't know if this is intended since I get no warnings and design verification passes normally.

 

The board isn't broken, I tried a different PYNQ-Z2 board and got the same result. I'm probably doing something terribly wrong since I'm still a beginner with AXI stuff.
Thanks in advance.

design_1.pdf design_1_addressing.pdf main.c

[asmi]

37 minutes ago, Embargo3198 said:

1. AXI peripherals only work if their base addresses are 0x4000_0000, 0x4200_0000, 0x4400_0000 or 0x4600_0000

Anything other than that, be it 0x4100_0000, 0x4001_0000 or 0x4800_0000, just doesn't work and both the PS and RISC-V core won't read or write to those addresses. The PS blocks on reading, which I confirmed when debugging and I'm not sure what the core does.

The PS writing directly to AXI GPIO which controls the onboard LEDs works fine when its address is 0x4600_0000 but not when it's 0x4800_0000.

That doesn't make any sense. Try getting rid of RV core and see if problem persists, then you will know what the problem is.

38 minutes ago, Embargo3198 said:

For some reason, the AXI GPIO registers I'm writing to from the PS's second AXI GP master port through "cpu_ctrl_interconnect" work fine with packed addresses. Although, just swapping 0x4000_0000 for 0x8000_0000 and the PS's AXI ports yields the same result.

You do know that each of GP ports has it's own address space, and attempting to access any addresses outside that space will never reach the port? See address map in UG585, Ch 4.1

 

43 minutes ago, Embargo3198 said:

2. Third AXI memory controller/BRAM generator pair doesn't work

43 minutes ago, Embargo3198 said:

3. When using AXI GPIO only the first channel is functional

Again, try getting rid of RV and see if problem persists.

Alternatively, you can use a debugger to inspect live transactions on AXI interface(s) to see what the heck is actually going on.

 

44 minutes ago, Embargo3198 said:

4. The AXI BRAMs are limited to 32 kB
When choosing more than 32 kB, for instance 64 kB, only the first 32 kB (up to base + 0x7FFF) are usable and writing or reading from it doesn't work and blocks the PS indefinitely. I don't know if this is intended since I get no warnings and design verification passes normally.

You did update the "range" column in address editor for a wider address window, right?

[Embargo3198]

39 minutes ago, asmi said:

You do know that each of GP ports has it's own address space, and attempting to access any addresses outside that space will never reach the port? See address map in UG585, Ch 4.1

Yeah, what I meant is, I tried the same thing but with the ports/interconnects swapped i.e. 0x4000_0000 for cpu control and 0x8000_0000 for memory access.
 

43 minutes ago, asmi said:

You did update the "range" column in address editor for a wider address window, right?

That's the only way to do it, no? Since the settings in the controller and BRAM generator are greyed out.

I'll try removing the RISC-V core and debugging some more.
If it is indeed the core, how does that even happen? It's held in reset until the command to enable it is sent.

Thanks for your help!

[asmi]

39 minutes ago, Embargo3198 said:

If it is indeed the core, how does that even happen? It's held in reset until the command to enable it is sent.

I don't know - that's what debugging is for. I would begin with inspecting an AXI bus from that core in a debugger just to see what's going on there. Also the kind of syncronization you use is not the best - I would use a mutex or a spinlock with atomic operations on both sides, instead of constantly resetting the core. Ditto for using AXI GPIO to control internal signals - I would create a simple custom AXI-lite slave with a single register. Thought I know some people prefer using PS GPIO ports 2 and 3 via EMIO as they give up to 64 channels for zero logic cost (you wil probably need to resyncronize it in PL as PS GPIO is likely running on a different clock from whatever you've got in PL - but that of course is only important for syncronous signals).

Edited March 29 by asmi

[Embargo3198]

2 hours ago, asmi said:

Also the kind of syncronization you use is not the best - I would use a mutex or a spinlock with atomic operations on both sides, instead of constantly resetting the core.

It's not for synchronization, the PS is waiting in a loop while the core runs. I just had problems with the core locking up the bus if I manually stopped it in the middle of an AXI transaction so now I just reset the whole network.

Thanks for the tip on the signals!

[Embargo3198]

21 hours ago, asmi said:

That doesn't make any sense. Try getting rid of RV core and see if problem persists, then you will know what the problem is.

I removed the core and things still didn't work.
Then I started over with the 2023.1 version of the tools and the GPIOs and memory work perfectly without the core.

Also the issue with AXI GPIO was that it wasn't being configured correctly for some reason. Even though I configured the second channel in Vivado, the GPIO2_TRI register still had the wrong data at boot and manually writing to it fixed the problem.

Again, thanks a lot for your help!
Now I just hope adding back the core doesn't break everything...