Zybo: Linaro/Archlinux and physicall memory buffers

[Alejandro Wolf]

Hi.
I managged to run both Linaro and ArchLinux on the Zybo board, with HDMI output (but i'm gonna remove it after). But I'm having some troubles to comunicate with my custom hardware.

I'm used to programming in Hardware, and I made several real time high performance applications like face recognition and real time video processing on pure verilog (Nexys 3, Nexys 4, Atlys boards), but I'm kind of new to cross-compiling, linux kernels, and drivers. I have a few master AXI modules that directly communicates with the DDR3 memory using the HP ports (no DMA, VDMA or similiars), and I just reserve a fraction of the DDR3 memory to work with them. Now in Linux, all the address are virtual, so I must write a driver, or find a way to allocate physicall contiguous memory in software.
-My IP usually work this way (e.g this is a sobel filter): It receives the base address where the image to process is located (usually is XRGB888 format), the resolution of the image, several arguments like threshold, and it outputs directly the result on the VGA port or HDMI port or a touchscreen that I have (all of them are custom IP) or it write the output on the DDR3 memory. If the flow requires SW interventions, It triggers a interrupt.

Now that I'm using Linux, I have 3 options: 

1. Use DMA: I kind of know how to use DMA on baremetal on the software side, but I have no idea how to make a custom IP that comunicates with the DMA modules. I have a decent understanding how master and slave AXI full interfaces works tough.
2. Write a custom driver that works the way my IP needs.
3. Hack with things like dma-mapping, contiguous memory allocations, memory maps, etc.
4. Make a "don't touch" region of ram for Linux, so the HW can work with it.

With simple AXI peripherals that are mapped to a fixed address (0x4XXXXXXX) and doesnt communicate with the DDR3, I use mmap and to access the registers, and it work well, but I can't use interrupts. With peripherals that use the DDR3, the configuration parts is done the same way that the regular AXI peripherals, but the RAM buffer, needs to be reserved so Linux doesn't write to it, or it only do it on my command. Also I need the physicall address of that reserverd ram region to pass it to the peripheral.

I'd like some highlights on how to make a custom DMA module (hardware side please), but any of the 4 options works. 

PS: Also, some of these modules that work with RAM doesn't even have an slave axi interface, so they won't appear on the automatic generated device tree.

Any information would be usefull. 

Best regards.

Alejandro Wolf.

[jamey.hicks]

Hi Alejandro,

• Use DMA: I kind of know how to use DMA on baremetal on the software side, but I have no idea how to make a custom IP that comunicates with the DMA modules. I have a decent understanding how master and slave AXI full interfaces works tough.

By "Use DMA", I assume you mean the central DMA controller, since you already have AXI masters that can read and write system DRAM via the HP ports. I do not think this approach will solve any problems for you, because you still would need a way to allocate memory to use as buffers.

• Write a custom driver that works the way my IP needs.

We wrote a couple of fairly generic device drivers. One simplifying assumption we made was to add address translation to our hardware. It's pipelined and it is not very big, but it's implemented in Bluespec System Verilog so you might not be able to use it. However, the design is simple, so I do not think it would take you long to implement something similar.

The MMU is simplified because portalmem allocates the largest physically contiguous regions it can, starting at 2^8 pages, then 2^4 pages, then single pages.

The first driver we have is portalmem, which enables user-space programs to allocate dma-bufs. Each memory region allocated has a corresponding file descriptor, so the regions can be passed between processes atomically and memory-mapped. Also, the file descriptor can be passed to an FPGA device driver, which can get the virtual->phys mapping for the region as a scatter-gather list.

The second driver is zynqportal, which passes the scatter-gather address translation info to the hardware. It also enables user-space to mmap the hardware registers and wait for interrupts via the poll() system call. This driver depends on just a few registers being implemented in the logic plus a command and response FIFO to interact with the MMU.

In case this is helpful, here are pointers to the drivers and the MMU:

https://github.com/cambridgehackers/connectal/blob/master/drivers/portalmem/portalmem.c

https://github.com/cambridgehackers/connectal/blob/master/drivers/zynqportal/zynqportal.c

https://github.com/cambridgehackers/connectal/blob/master/drivers/bsv/MMU.bsv

• Hack with things like dma-mapping, contiguous memory allocations, memory maps, etc.

I looked for a way to do DMA from user-space but did not find it. Did I miss something?

• Make a "don't touch" region of ram for Linux, so the HW can work with it.

I do not know how many memory regions you need to allocate, but the simplest way to get contiguous physical memory from the kernel is to reserve it off the top, passing "mem=xxx" in the kernel cmdline, where "xxx" is less than the amount of memory on the board.

 

[Alejandro Wolf]

Thanks Jamey.
I'll try those options.

PS: I never writed a driver before. The examples that I see are already "done". I would love an step-by-step guide on how to write drivers.

best regards

Alejandro Wolf

[Cristian.Fatu]

Hello,

Have you considered using a framebuffer driver ? Writing it is pretty basic, using it under Linux is easy, the only problem is that it offers limited video functionality.

http://tldp.org/HOWTO/Framebuffer-HOWTO/

Then, if you can use the framebufer, Linux will "take care" of populating the memory region that you specify when you write the driver with the video content.

Regards, 

Cristian Fatu

[Alejandro Wolf]

Hello,

Have you considered using a framebuffer driver ? Writing it is pretty basic, using it under Linux is easy, the only problem is that it offers limited video functionality.

http://tldp.org/HOWTO/Framebuffer-HOWTO/

Then, if you can use the framebufer, Linux will "take care" of populating the memory region that you specify when you write the driver with the video content.

Regards, 

Cristian Fatu

​Thanks Cristian. Maybe that will cover most of my needs. I'll look into that.

[Alejandro Wolf]

 

I do not know how many memory regions you need to allocate, but the simplest way to get contiguous physical memory from the kernel is to reserve it off the top, passing "mem=xxx" in the kernel cmdline, where "xxx" is less than the amount of memory on the board.

​If I use mem in the bootargs the system halts in "starting kernel"

I guess it is because the device tree copies for some reason to the top of the memory, and that address is removed. Here is the output
 

U-Boot 2014.01-00005-gc29bed9-dirty (May 11 2015 - 12:22:58)

I2C:   ready
Memory: ECC disabled
DRAM:  512 MiB
MMC:   zynq_sdhci: 0
SF: Detected S25FL128S_64K with page size 256 Bytes, erase size 64 KiB, total 16 MiB
*** Warning - bad CRC, using default environment

In:    serial
Out:   serial
Err:   serial
Net:   Gem.e000b000
Hit any key to stop autoboot:  0
Device: zynq_sdhci
Manufacturer ID: 1b
OEM: 534d
Name: 00000
Tran Speed: 50000000
Rd Block Len: 512
SD version 3.0
High Capacity: Yes
Capacity: 14.6 GiB
Bus Width: 4-bit
reading uEnv.txt
** Unable to read file uEnv.txt **
Copying Linux from SD to RAM fs...
reading uImage
3459296 bytes read in 304 ms (10.9 MiB/s)
reading devicetree.dtb
13495 bytes read in 16 ms (823.2 KiB/s)
## Booting kernel from Legacy Image at 03000000 ...
   Image Name:   Linux-3.18.0-xilinx-46110-gd627f
   Image Type:   ARM Linux Kernel Image (uncompressed)
   Data Size:    3459232 Bytes = 3.3 MiB
   Load Address: 00008000
   Entry Point:  00008000
   Verifying Checksum ... OK
## Flattened Device Tree blob at 02a00000
   Booting using the fdt blob at 0x2a00000
   Loading Kernel Image ... OK
   Loading Device Tree to 1fb29000, end 1fb2f4b6 ... OK

Starting kernel ...

 

[Alejandro Wolf]

I resolved the map=xx issue by modifying the u-boot source.

I had to change fdt_high and initrd_high in /include/configs/zynq_zybo.h so the u-boot wont place anything on the top of the memory