Help for Arty A7 MIG read

[escou64]

I am currently trying to implement an interface between multiple ports of a custom memory bus and the MIG IP of my Arty A7-35T board.Basically in my current example design, the first port reads a memory range 128-bit by 128-bit (it is for a custom VDMA). In the same time, the second port writes values in the same memory range (it is to emulate a CPU activity). The first port has the priority: if it is not busy, it sends its read request and else, its the second port which tries to write. I try to debug my design with an ILA.

(See picture normal_read) In the first cycles after the reset, the system looks to work as expected. The app_rd_valid is asserted some cycles after the request (r_rpend indicates how many read requests are pending, r_mig_not is simply used to try to visualize the clock, r_mig_cycle indicates the number of cycles since the last reset).

But later during the execution, the whole system seems to be locked (or slowed down). Particularly, the MIG becomes really slow during read requests. We can see (on pictures rpend_increase and long_read) that app_rd_valid is now asserted only after many cycles (512+ cycles if we consider the r_mig_cycle[15:8]) while the app_rdy is high. This hypothesis is validated by the value of r_mig_rpend which stays at 7 most of the time ... Finally, it seems that the MIG does not respond at all to my read requests since the trigger on app_rd_valid is never activated again.

I thought that the MIG asserts a read request in only some cycles (~22 cycles). Is that correct ? Do you have any ideas of what conditions I have possibly missed which leads to this strange working ? Or a possible strategy to debug my design ?

Thanks for your help!

[zygot]

I suppose that you've looked this post over: https://forum.digilent.com/topic/22197-a-guide-to-using-ddr-in-the-all-hdl-design-flow/

Debugging a complex design using the ILA is going to be a slow and arduous sled ride.

First, you've gotten a basic design to work in a manner similar to what the tutorial presents, yes? If not that's the place to start.

The Mig IP nice because it gives you all of the sources in Verilog. It and the example project, unfortunately, are written in as obtuse and difficult to rost out manner as possible. Trying to turn it into a multi-channel DDR controller is going to be difficult. The first phase of pain will be to simulate the working vanilla Mig-based design as a baseline. You will have to do your design in Verilog. If you can't work out how to do that step, then you are in for some hard times. This is the only rational way to know if your own Verilog modules are working as expected.

If you can't do that step then you will have to slog through the IP code and instrument it. Starting from something that doesn't work is most likely going to take more time than starting with something that does work and that you can enhance bit by bit.

If you are interacting with the Mig controller properly I wouldn't expect to see the kinds of behavior that you are experiencing. That kind of points to your VDMA code. The Mig has a debug feature that you might want to enable. This may or may not help. I suspect that you are expecting the DDR controller to operate in a linear manner, and that is probably not what it is doing. If you aren't managing all of the read/write bytes per burst operation this would lead to problems. If you aren't monitoring fault or error signals deep within the IP then it's going to be hard to figure out what's going on using an ILA. The ILA isn't ideal for all debugging needs. Sometimes you have to create your own LA IP to capture elusive events and present them in a form that you can understand.

[escou64]

17 hours ago, zygot said:

Quote

I suppose that you've looked this post over: https://forum.digilent.com/topic/22197-a-guide-to-using-ddr-in-the-all-hdl-design-flow/

Thanks for you answer: I'll take a closer look on this post to make sure I haven't missed anything.
 

Quote

First, you've gotten a basic design to work in a manner similar to what the tutorial presents, yes? If not that's the place to start.

The Mig IP nice because it gives you all of the sources in Verilog. It and the example project, unfortunately, are written in as obtuse and difficult to rost out manner as possible. Trying to turn it into a multi-channel DDR controller is going to be difficult. The first phase of pain will be to simulate the working vanilla Mig-based design as a baseline. You will have to do your design in Verilog. If you can't work out how to do that step, then you are in for some hard times. This is the only rational way to know if your own Verilog modules are working as expected.

Just to to be sure that I am clear on my issue: I'm not trying to redesign a more complex MIG. I am just using the one available on Vivado which has one use interface port. It's in my own system where I have a module which receives requests coming from two ports and tries to redirect them to the one MIG port. It acts like a "multiplexer" with priorities. That's why I am surprised when I see the MIG not responding after many cycles while it seems to work fine in the beginning of the execution ...
 

Quote

 

Debugging a complex design using the ILA is going to be a slow and arduous sled ride.

If you can't do that step then you will have to slog through the IP code and instrument it. Starting from something that doesn't work is most likely going to take more time than starting with something that does work and that you can enhance bit by bit.

If you are interacting with the Mig controller properly I wouldn't expect to see the kinds of behavior that you are experiencing. That kind of points to your VDMA code. The Mig has a debug feature that you might want to enable. This may or may not help. I suspect that you are expecting the DDR controller to operate in a linear manner, and that is probably not what it is doing. If you aren't managing all of the read/write bytes per burst operation this would lead to problems. If you aren't monitoring fault or error signals deep within the IP then it's going to be hard to figure out what's going on using an ILA. The ILA isn't ideal for all debugging needs. Sometimes you have to create your own LA IP to capture elusive events and present them in a form that you can understand.

 

I am not sure to understand how I can debug the design without the ILA here: simulate my own design with MIG seems completely impossible and I do not have any other way to observe signals during execution ...

However, I did not try to use the debug feature of the MIG. If I understand correctly, it adds some signals to the MIG interface (dbg_*) which allow to view internal states of the MIG. Am I right ?

 

[zygot]

Yes, I understand what you are trying to do. For background, the Spartan 6 devices had a hard external memory controller supporting multiple channels. When Xilinx abandoned the hard controller in the Series 7 family it also abandoned the multi-channel external memory model, now implemented in logic. That might tell you something. I believe that there might be a path to a multi-channel external memory controller using AXI IP, which is completely at odds with an all HDL design flow in my opinion. I haven't explored this avenue. In theory you might be able to achieve your VDMA goals but it's likely a bit harder than you imagine. That's why I suggest starting off with a basic design project using the Mig IP as at start. In my experience, vital knowledge is gained by struggling with incremental changes in design complexity. Knowledge isn't just what you read; it's how you connect the basic details conceptually, how you understand the subtle interactions and minutiae of what's going on in your logic. You can't get that by skipping ahead to the final result for complex designs. I'd put DDR in the complex design category. The content in your post tells me that you are stuck. To me that says that your experience and debugging skills haven't quite prepared you for this project. Sometimes you can get lucky and solve problems without having an intimate knowledge of the details of the inner workings... usually you don't get to be lucky.

Simulating a design using the Mig IP is certainly not impossible. It is a vital skill however, just hard to do due to the way that the IP was implemented. I do encourage you to simulate a basic design usiing the IP. The fruit of the effort will be more than worth the time and energy that it will take. Do use Verilog for your toplevel module if you want to succeed in simulating the Mig.

You can read the Mig IP user manual for details about it's debug facility to determine if it's worth the added complexity in source code and implementation.

The ILA is a quick and relatively painless way to peek into your hardware as it operates. It's also very limited in triggering capability to capture specific events related to complex logic states. It's amazing what a state machine, a FIFO and an a UART can to to aid complicated debugging that the ILA can't or has a hard time with. Vivado debug facilities can become unusable debugging some designs. I know this from experience.

I realize that you see a fairly simple and straight line for accomplishing your project goals. This might be because you lack the comprehension and understanding of how the logic actually operates. Start with a careful reading of the Mig IP user's manual to get a sense of what the moving parts are. Gain skill and knowledge by starting with someting that works and progressing toward your ultimate goal. Broaden range of tools in your debugging toolbox to include simulation and custom debug IP that you've written and tested.

The tutorial isn't completed byt the way. I haven't yet included the simulation part. I've simulated Xilinx Mig designs but haven't worked out how to present the information the way that I want suitable for rest of the tutorial. It's complicated and not well documented. One hint is to have an all Verilog design and testbench. Another is to short-circuit the calibration phase as this encompasses too much simulation time to the point where the simulator is too slow to be functional. Some knowledge is available in things that you can read and some knowledge only comes from doing, failing, figuring out an way around obstacles and 'rinse and repeat until success'.

[jb9631]

On 6/30/2022 at 4:43 PM, zygot said:

Yes, I understand what you are trying to do. For background, the Spartan 6 devices had a hard external memory controller supporting multiple channels. When Xilinx abandoned the hard controller in the Series 7 family it also abandoned the multi-channel external memory model, now implemented in logic.

Hello again after a long while, friend. I'm sorry for derailing another thread, I promise to keep this one short :)

The hard external memory controller you mention is a strange concept, on two parts. First, I've read over XAPP721, a Xilinx note dealing with a DDR2 interface on the Virtex-4. It's a fascinating read that is still somewhat relevant in the age of the 7-series chips, as the primitives used--most notably the SERDES and IDELAY--are still around, and can be made to function with DDR3 (having managed to build a working PHY + controller from scratch for my Arty S7-50 -- not that I'd ever recommend doing this, but we discussed this once already, when I was only just starting). It's peculiar that Xilinx would go from an implementation in logic to a hard controller, and back to logic again through series 4->6->7. Second, if this Spartan-6 hard controller was anything like the more modern 7-series MIG, there's a whole host of modes and/or primitives that simply aren't documented anywhere in Xilinx's datasheets or application notes (as e.g. MIG uses phasers, but they aren't explained in docs like UG471 beyond noting their existence is reserved "for use with MIG"). Much of the modern MIG core is a black box, at least as far as the PHY is concerned.

On 6/30/2022 at 4:43 PM, zygot said:

That's why I suggest starting off with a basic design project using the Mig IP as at start. In my experience, vital knowledge is gained by struggling with incremental changes in design complexity. /.../ To me that says that your experience and debugging skills haven't quite prepared you for this project. Sometimes you can get lucky and solve problems without having an intimate knowledge of the details of the inner workings... usually you don't get to be lucky.

Simulating a design using the Mig IP is certainly not impossible. It is a vital skill however, just hard to do due to the way that the IP was implemented. I do encourage you to simulate a basic design usiing the IP. The fruit of the effort will be more than worth the time and energy that it will take. Do use Verilog for your toplevel module if you want to succeed in simulating the Mig.

If @escou64 would like another example of using the MIG core, I would be happy to provide them my own application. It's simple, but it's worked for me (though, granted, my tests were only up to 30 minutes long, and didn't do any sequential accesses). If you ask me, nobody is ever prepared to use the MIG without stumbling. (On this note: just today I found Dan's--of ZipCPU fame, also a poster on this forum--attempt at deciphering the workings of MIG and DDR3 on OpenCores, and even he had many difficulties, and I'd consider him to be pretty knowledgeable in the realm of FPGAs.) For me, the example MIG project that Xilinx provides is unreadable. I will agree it might be prudent to explore, or perhaps even to delve into the MIG's code, if the project at hand is a serious implementation that will be used and worked on for years to come, but I found it simpler to handle it as a black box with such-and-such inputs/outputs that demand such-and-such operation to work properly. Maybe that's just because I didn't need the core to be robust, or intended to work with it beyond testing it out; I just had to be certain that the Arty board I have on hand could communicate with the included DDR3 chip.

One note about Xilinx's documentation: After months of work and reading Xilinx's application notes and user guides, I must warn that some information might be missing or outright wrong in the manufacturer's own documentation. Most of it is fine, but some crucial details might not be (in my case specifically, there was one such instance regarding an input pin to the ISERDESE2 and OSERDESE2 primitives), and you'll go mad before you figure out that it's not your fault something doesn't work.

On 6/29/2022 at 7:17 PM, escou64 said:

Do you have any ideas of what conditions I have possibly missed which leads to this strange working ? Or a possible strategy to debug my design ?

Back to the original poster: I, too, can vouch for zygot's tutorial. I didn't find the need to fully complete it before making my MIG design work, but it's a quality write-up worthy, in my opinion, of more than "just" a digilent forum post. (God forbid this website is ever deleted!)

To address your question specifically, I happen to have had a very similar issue: Make sure the FPGA chip you have can drive the frequencies you are requesting from it. I had a similar problem, with MIG refusing to work after a short runtime. I had been running the MIG core at 325 MHz (650 MT/s) in 2:1 mode. Eventually I stumbled upon the relevant timing datasheet for my chip (FYI: it's DS181 for Artix chips) and there I found "Table 16: Maximum Physical Interface (PHY) Rate for Memory Interface IP available with the Memory Interface Generator" (might be named slightly differently in your relevant document). In 2:1 mode, the MIG could only drive my DDR3L SDRAM at 620 MT/s (below my 650 MT/s requirement). I had to reconfigure MIG in 4:1 mode, where the chip is noted to be capable of 667 MT/s. This might be tangentially relevant to your question, or it might not apply at all; I haven't seen this behavior documented anywhere, but I experienced it firsthand. Going to 4:1 solved this bug where I could read and write a little, but then after a few minutes maybe, the core would stop responding.

Also, documentation on the 2:1 MIG mode is sparse to begin with. Digilent's own reference manual was of little help, as it instructs the user to change a few options, but leaves most of them untouched, or "default". Apparently, between somebody at Digilent having written the RM and today (or rather, H2 2019, as that's the version of Vivado I'm using), some of the options in the MIG configurator have changed their default values. As luck would have it, the default in my case is the 2:1 mode, in which my Arty board cannot function.

--jb

Edited July 9, 2022 by jb9631

[zygot]

@jb9631

15 hours ago, jb9631 said:

I'm sorry for derailing another thread

I don't think that your post fit the category of derailing, or even hijacking, the thread started by @escou64. Most if the content is directly applicable. Some are tangential... but still relevant. Personally, I find that the tangential can be more informative than people tend to realize. Usually, people posting to a site like this one have a single-minded desire to get past a wall blocking them from completing a project, and have no interest in learning about the details, much less the tangential ones. But it's those tangential bits of knowledge and perspective that make dealing with the next hard challenge a bit easier to tackle. So, based on your post, here are a few thoughts that may be worth reading, or not, depending on how you approach FPGA development.

• Historical context can be very informative and helpful in dealing with FPGA vendor tools and IP that are hard to decompose. The vendors would rather have customers that are only interested in completing the task before them now, preferably using the handful of vendor IP that the tools offer, not concerned with the details of how they work. There's nothing wrong about wanting the tools to do your work for you, and not caring about learning details... as long as the tools get the job done. If you are going to spend a lot of time doing FPGA development, particularly using the 'advanced' features of the devices, then you are likely to get idisabused of that model very quickly. If you understand the features of those old Spartan 3 devices, it's a pretty fascinating journey comparing those ancient families to the most recent ones. I have no doubt that implementation design improvements and FAB node evolution has made the basic clocking and IO features better, in most ways. But not better, necessarily, for all customer needs.
• I don't just use one FPGA vendor. It's interesting how the two giants in the market have copied what the other does, both in terms of device capabilities and features, and tool experiences. There was a time when Altera tools were so much more stable and bug free, compared to its rival, that many companies selected programmable devices on the merits of the tools alone. Those days are gone as all of the software that makes the tools has gotten too complex to manage. There are two spheres of reality concerning the tools. There's the stuff that the customer can see, that is the GUI, the tcl, pearls and other scripts that setup and create the vendor IP and connect all of the sources together, and the executable software where the 'magic' happens and sources are turned into bitstreams.
• In my lengthy experience,regardless of the silicon device documentation in the way of datasheets, application notes, and such have always been a bit short of details, or just plain wrong, when the customer is faced with difficulties using a device for a particular purpose. Undeniably, part or most of this is due to how difficult it is to write, much less maintain documentation that's error free. Also true, is that this is an opportunity to game the customer experience with vendor devices in favor of the vendor. I won't elaborate because this discussion is more worthy of book writing than forum posting. My point is that sometimes one just has no option other than embarking on a quest for the holy grail, which in this context means understanding details and conducting an extensive hunt for relevant information that separates the nonsense from the necessary facts. There is no 'conspiracy theory' at work here... just the unfortunate side of how people and companies make money and compete in the marketplace.
• I mentioned that one facet of programmable logic that the customer can't ever parse, the inner workings of the executable(s) that grind out loadable bitstreams used to configure programmable logic devices.. that is the tool software. It's always been the case that what goes on behind the curtain isn't exactly what the documentation and all of those IP, timing and project related scripts would suggest. If one wants to use the advanced features of UltraScale devices, this becomes all the more clearer. A lot of control and implementation details are really done in the tool software. This most certainly puts the advantage in favor of the vendor in terms of what a customer can do with vendor devices and, in the end, how income gets generated. In my experience there has always been, for certain projects a disconnect between what vendor documentation says and actual implementation experience when using some device features. With UltraScale this boundary between what the documentation says about how certain features are implemented and what's actually controlled behind the curtain is just more apparent; never mind the trend to more encrypted sources and less control for the user. In the case of the other major FPGA vendor there's no embarrassment about using the vendor documentation, tools and scripts to 'encourage' customers toward a path that results in greater dependency on the part of the customer and a greater income stream for the vendor. All of these things are interconnected.

I'm a bit less capable of being succinct, but I do think that for some people there's a lot more to consider than just being able to use vendor supplied resources to complete a particular objective for doing programmable logic development effectively and efficiently. Hopefully, my thoughts are worth thinking about. If not, you've probably not gotten this far and that's OK with me. What's boring or meaningless to one person might be a helpful spark of insight to another.

Edited July 9, 2022 by zygot