Video RAM in DDR

[Dannny]

Hi,

I want to develop a simple subsystem that is shown on the picture below. Basically, I need one producer that would generate a picture(s) and one consumer that would display the picture. The consumer is easier part, it's VGA block that reads the pictures from the memory. I want to use a bigger resolution, so I cannot use BRAM and I have to go for DDR. I'm planning to use MIG for this job. The question is whether I can configure DDR as a dual-port ram (one port for generator and one port for consumer). Is it possible? If not, I think that I have to go with multi-master bus, so the consumer is another master on the bus... Other memories that Nexys A7 has are too small (and I guess also to slow).

Any help is appreciated.

Thanks, Dannny.

[zygot]

@Ciprian,

I understand why there's a desire to stuff Linux into some FPGA designs. In fact you've fleshed out a few of the difficult problems that someone taking the HDL flow must solve on their own quite well for me. @Dannny has discovered one of the other conundrums which is trying to fit an idea into an FPGA platform as opposed to choosing an FPGA platform appropriate to your idea. Sometimes the choice of external memory designed into a board limits comes with unforeseen restrictions. You've also nicely exposed another problem in that even if you can get a non-ZYNQ platform running some form of Linux that doesn't necessarily resolve your problems. Furthermore, using the limited IP available with the MiccroBlaze comes with a problem that only appears after many weeks of work. What if you need your DDR video buffer to work differently or better? Then what? Likely, you have to do all of the stuff that you correctly point out is very hard to do.

I don't know what the purpose of the design exercise is. If I had to execute a design that leveraged open source Linux source material I'd certainly want to do it on a ZYNQ platform. But the typical Z7000 family FPGA board is not a wonderful Linux platform  if you need performance. Creating a custom Linux build reasonably scaled to the resources of you FPGA platform is not easy. If you are trying to work out how a processor driven DDR video frame buffer might be implemented you will end up knowing a lot more doing it yourself then relying on IP that you don't understand or can't easily modify.

There's nothing wrong with slapping together a quick MicroBlaze design using available IP from your FPGA vendor, if learning how to do that flow is the goal or if all you want to do is get to a point where you've accomplished something. Having something that you can adapt to any FPGA platform or project is a different matter. Developing something that you can easily rework or improve is another matter, except that you don't realize it until you've invested a lot of time into getting to the point of having a design that you hope can be a springboard into something better.

Hopefully, having some idea of the obstacles  that lie ahead will help make for good early design decisions. I know, it's complicated.

 

[zygot]

On a slightly higher level of conceptualization, for something like a video buffer. I do want to point out that a lot of solutions to FPGA design problems have nothing to do with programmable logic or HDL or development flow. While being given a relatively easy path to constructing complex structures may get you so far... they also only get you so far, and are limited in how you can use a very limited set of structures. Better to learn about video, and older ways of implementing video and constructing those structures. There's a danger to having a false sense of accomplishment and thinking that you've overcome a particular design obstacle, when in fact you've just used a borrowed toy that you can't keep. Like most things in life, the quality and usefulness of things that you acquire are directly tied to the amount of work and effort put into obtaining them.

Perhpas. one day, the only source flow for FPGA development will be an AI that you say " Hey Vivi, I'd like to do.....". I for one, am not waiting with eager anticipation for such a day when man is dumber than his robotic creations or can't imagine without them.

[zygot]

@Ciprian,

I'm glad that you decided to add a bit more shading to a very important philosophical concept. It's hard to capture all of the important facets in one post reply. While you are correct that I do advocate for what is a  harder, slower and more demanding design source choice in going all HDL, I do understand that there are projects and people for whom that might not be the best choice.

No one know how hard it is to maintain a demo so that customers can replicate a project over the course of even a few Vivado revisions than the fine people at Digilent. A high percentage of posts to the FPGA forum revolve around user's inability to merely build demos created in older versions of Vivado. What I can say with certainty on that subject is that I have HDL code that I wrote 20 years ago, not even for a Xilinx target device, that I can still use today. No hassle, just add the source to my current project. True, some use basic resources like block memory or clock generators that are implemented differently from device family to device family or vendor to vendor, but even those are pretty simple to change. A recent version of Vivado broke all HDL source code using its own basic resource IP. But fixing the issue isn't a major headache. As long as I record notes about the design process and document my code well I never lose the IP that I write, nor the understanding of how to address a design problem. If your design relied on IP that was free and was changed to requiring a license there's only only one option, which is to keep an older version of Vivado around to use the 'free' IP. The quotes around free are there because free always has a cost in one form or another.

The 'free' IP and board design flow can certainly produce a result in a short period of time but it also comes with not so obvious strings attached. Some of these strings you've pointed out. The truth is that it takes time and commitment to get to the point where anyone can master FPGA development to the point where you can have the confidence to tackle complicated projects with all HDL source code. My argument is that the board design approach entices novices into thinking that you can do substantial FPGA development without the HDL skills. I say, after you have the skills then perhaps you are in a position to evaluate whether or not the 'free' IP is worth the cost. I liken the board design flow to a walled garden, but perhaps a sandbox is a more appropriate analogy. Sure, there are things that you can do without much knowledge or hassle in the sandbox but eventually you will get bored with the limited toys available to you and need to add your own HDL anyway. And no one like having their limited selection of toy taken away form them once they are relied on. To my way of thinking, the hard arduous path to working with programmable devices is the inevitable path if you are going to keep doing FPGA design for any length of time. If you have any interest of doing FPGA development professionally, then you have better have the skills and experience in figuring out how to find your own conceptual design solutions and strategies for debugging designs and verifying designs. The board design approach provides little or no support for the really important skills needed to be self-sufficient.

I completely agree with the premise that it is up individuals to decide what design source flow is best for them or their project. And considering all points of view and the experiences of others is an important part of that decision making process.

 

 

[Ciprian]

Disclaimer: I considered this Thread closed before writing this post, since Dannny has gotten his answer.

Normally I would not comment on this thread anymore because it seams to have concluded, but its so rare to find a naturally developing "philosophical" dilemma in engineering circles that it's hard not to comment on it. I'd also would like to point out that this might help @Dannny in his current and future projects.

The way I see it, the dilemma is similar to building a carriage. Zygot and D@n are strong advocates of the build it yourself philosophy, this means build your own wheel, build your own axie, seats, shafts and everything. I'm more of the design you carriage based on the components you have at hand, go to the wheeler and buy a "standard" wheel, find a widely used axie, seats and shaft and then customize it using those components.

Their way of doing things is arguably better then what I do, it forces you to learn as much as you can about all aspects or you project and because you have written it you are less depended on others  with it and what is more important it is yours. It, unfortunately, is also very time consuming and if you don't adhere to a standard you will not be able to find help as easily, also, and this is the worst part, you are in charge of maintaining all the pieces of your project (updates, drivers, incompatibilities). This is where zygots diver comes in to play:

On 7/3/2020 at 7:06 PM, zygot said:

Another thought on Linux as a platform. I have written a loadable kernel driver. It works fine on my kernel 2.6 Centos 6.2 box but is totally useless in Linux distributions with a 'modern' kernal. The sad reality is that i technology everything's a moving target. Usually that's good, sometimes that's really bad.

The way I like to do it, is more "hacky". You are in charge mostly of finding the right components for your design and make them work together. You save time on building everything yourself... if you lock yourself in using a certain bus, like AXI or Wishbone, you might find that there are components available for it, which people are actively maintain and regularly update if needed. The main issue is that there is no guarantee, it might work it might not, bugs are out of your control, maintenance might stop at one point and stuff becomes out dated or obsolete. You will learn about your project but you will not know every little detail, aspect or snippet of code in your design which makes it a black box in certain respects. An example for this the Test Pattern Generator by Xilinx. Up to Vivado 2016 (I think, can't remember the exact version) it was free to use, afterward they just moved it to payed lic. and if you want to use it then you need to pay for it.

A very important aspect here is also the licensing of the IPs, if you build it yourself you own it and you can do with it as you please, if you take somebody else's code you will need to take a close look at the licensing agreements to figure out you limitations. Specially  if you want to manufacture stuff with it.

Ultimately it comes down to time, resources, and support. You will need to chose which is more suitable for your project and what your end goal is. In my opinion, because @Dannny wants to use his design with linux as a final goal, he should try to stick to a proven concept which has support for linux as well before he goes down a path of frustration and hard work. Because the linux community is helpful but you need a lot of knowledge  in linux to know how and where to ask for help. There are not many who do embedded linux on FPGA...

-Ciprian

[zygot]

Another thought on Linux as a platform. I have written a loadable kernel driver. It works fine on my kernel 2.6 Centos 6.2 box but is totally useless in Linux distributions with a 'modern' kernal. The sad reality is that i technology everything's a moving target. Usually that's good, sometimes that's really bad.

I cook with pots that were my mother's in the middle of the last century. Even if I could find anything nearly as well made and good for cooking I couldn't afford to buy them. That's a rather silly example of bad technological advances.

[zygot]

43 minutes ago, D@n said:

A "Video DMA" or "VDMA" is just a piece of RTL that copies memory from block RAM to a video stream

Yeah, that's true. All of Xilinx's IP is, at its source just RTL. So is your Intel processor in your PC that runs Vivado.

DMA can sound more impressive than it is. Recently, I posted a Project Vault ZYNQ design that has no AXI HDL code and a minimalist board design. It has a couple of 2Kx32-bit 'maibox' BRAMs that both the PS and my HDL can do read/write transactions. The HDL has a 'DMA Engine' that pushes data between the mailbox BRAM and the HD application BRAM. Don't be impressed by the fancy name; it's just an address counter in one process...

Be like @D@n  when he works on his own projects... don't use AXI in your HDL designs.

[D@n]

@Dannny,

A "Video DMA" or "VDMA" is just a piece of RTL that copies memory from block RAM to a video stream.  With a few exceptions, it's not all that hard to write.  As an example, I recently wrote a frame buffer to video stream DMA here.  You can find a demonstration of how that might be used to drive an HDMI signal here.  I figure it takes about 2 days or so to get the AXI interaction right--provided you have a bit of experience and a formal tool for verification.  As for @Ciprian's comments about the Linux kernel driver, I can't really comment.  I tend to do all RTL designs, and so I haven't really had much of a need for a Linux kernel driver.

I will caution you this: beware of bandwidth requirements on your DDR3 memory.  One of my earlier projects, using a 1080p/60Hz input and similar output, didn't have the bandwidth I wanted to go through memory in between the incoming image and the outgoing one.

Something to consider,

Dan

[zygot]

@Dannny, Hey, for what it's worth you've taught me something about the Mig that I haven't found out for myself yet. I guess that's the utility of having a forum where people can ask questions and get nswers from a variety of perspectives is all about. Everyone can learn things and get a little seed of inquiry implanted into the back of their minds. You never know what can happen.

[Dannny]

Thanks all of you. I was hoping that I can live without VDMA, but it seems that I have to do it anyway. The main reason for this exercise is to have fun and to learn something new. I found basic blocks for the embedded linux on github or opencores and I already tested some of them. So, I'm getting more and more units to work :-)... But the Video RAM is first "blocker" on the way. I'll look into other examples as well. If any of you have some other ideas, don't hesitate to share them :-)...

Thanks.

[Ciprian]

17 hours ago, zygot said:

[...]You can certainly implement the simple concept as outlined in your diagram using only HDL. 

Let's hold off on the final goal for now. It's not only OK, but a really good idea to run through a number of smaller design projects that can then be used to implement a grand idea. [...]

I respectfully disagree.

Firstly, it is very different to go from a HDL design to a fully functional Linux system. Mostly because by doing it this way you lock yourself in a path where you must create everything yourself, ultimately rewarding but highly time consuming. Then there is the fact that, given the final requirements, you want a solution which has Linux driver because writing a DMA or even worse a VDMA driver is a very, very big challenge.

The solution to @Dannny problem is a VDMA which has the functionality to use multiple frame buffers in the DDR or other memory devices. Further more it has Linux drivers and Bare-metal drivers. If your final goal is Linux then this is the safest way to go... What you need is a VDMA, VTC, AXI4S_to _video_out pipeline on the output path and Test pattern generator, VDMA pipe on the input... The whole thing doable and ultimately scalable to Linux with some effort, but much easier the creating in HDL and then writing drivers for it.

Although I don't have a full example for you I'm sure that you can figure it out if you focus on the VDMA and on the xilinx video solutions... For the output solution you can take a look at this, Zybo-Z7-10-base-linux.... it is for a Zynq and with HDMI, true, but the principle is still valid and ca be adapted for non-Zynq solutions.  There should be some examples on how to do this on the Xilinx forums or application notes, they might even have example projects... At Digilent we had some for Zynq, but I can't currently find them anywhere publicly posted.

This might also help: http://web-pcm.cnfm.fr/zybo-video-workshop-materials/

If you don't have 2017.4 I have attached the the Zybo-Z7-10-base-linux to this message. 

-Ciprian

Zybo-Z7-10-base-linux.zip

[zygot]

@Dannny,

Sorry about that. I don't have a Nexys A7 to play with so I didn't. The Spartan 6 family that the Artix replaces has a really nice multi-channel hard external memory controller as part of the deal. Sometimes progress is a step backwards...

Well, perhaps it's not time to jump ship. You can still use one channel to write/read the DDR. If you later decide that you want to add a soft processor this might be able to be addressed on a system level. There's almost always more than one way to get to a destination. You may have to do more thinking about how to get there using one channel.

[Dannny]

4 hours ago, zygot said:

[...] You also might decide that the first step would be to just implement a Mig with 2 controller channels and verify that you can do that, ignoring VGA. [...]

I just tried to use MIG to generate two controllers, but it seems that it's not suitable for the FPGA board. Vivado says that multi-controller needs DDR3 SDRAM whereas Nexys A7 has only DDR2 SDRAM...

[zygot]

1 hour ago, Dannny said:

I don't have to use MicroBlaze (or other pre-built blocks). In other words, I want to stay at HDL as much as I can.

Great! So Let's start off with a 'system' level design. You can certainly implement the simple concept as outlined in your diagram using only HDL. 

Let's hold off on the final goal for now. It's not only OK, but a really good idea to run through a number of smaller design projects that can then be used to implement a grand idea. So lets map out a system for testing your plan for loading some sort of image into DDR and displaying it on the VGA port. Take your diagram and add the parts that include everything that you think that you will need to do that. You also might decide that the first step would be to just implement a Mig with 2 controller channels and verify that you can do that, ignoring VGA. This would be easier and quicker.

Once you have  plan for how your step 1 test project is to be performed you can create toplevel HDL entity. Then you want to use the IP manager to create a native Mig external memory controller for your board. You will have to design your own read/write controllers to use the Mig. I'm mentioning this because there are quite a few details that, as an HDL designer, you will have to flesh out. If you look around you might be able to find some example code to help with implementing all of those 'black boxes'. That's the burden of doing HDL designs. The alternative is to let Vivado take care of most of the details in a board design flow, but then Vivado has control over what you can and can't do. Board design or HDL is an important first decision to make.  Yes, you an add HDL to a MicroBlaze board design but while you can get so far along quickly, progress can get quite slow as you add the final touches.

[Dannny]

25 minutes ago, zygot said:

Since you are using the Nexys A7 I think that the first question should be what kind of design source flow are you using. The Mig IP allows for creating multi-controller external memory designs so that's not a limitation if you are doing an HDL design. If you are using the MicroBlaze / board design then you have to work within what that provides. This approach will certainly consume more of your device resources. As I don't use MicroBlaze or the board design flow for non-ZYNQ projects I can't help with that.

I don't have to use MicroBlaze (or other pre-built blocks). In other words, I want to stay at HDL as much as I can. There are open source processors that can be easily put to Nexys A7 (either embedded ones or the ones with Linux support [e.g., OpenRISC, RISC-V processors, ...]). But this is the second step from my point of view. I'd like to get the video output figured out and then put a smarter image generator that could be a processor in the end (ultimately with Linux and framebuffer).

[Dannny]

2 hours ago, Ciprian said:

Hi @danny,

Before I can answer your question I need to know if you are planning on using it in an embedded linux or not. The solution might vary depending on this.

-Ciprian

The ultimate goal is to have linux with framebuffer (simplefb module) up and running. But I thought that I'll start with just a simple image generator to prove the concept first :-)...

[zygot]

Since you are using the Nexys A7 I think that the first question should be what kind of design source flow are you using. The Mig IP allows for creating multi-controller external memory designs so that's not a limitation if you are doing an HDL design. If you are using the MicroBlaze / board design then you have to work within what that provides. This approach will certainly consume more of your device resources. As I don't use MicroBlaze or the board design flow for non-ZYNQ projects I can't help with that.

[Ciprian]

Hi @danny,

Before I can answer your question I need to know if you are planning on using it in an embedded linux or not. The solution might vary depending on this.

-Ciprian