AXI Peripheral - User Logic testing

[engrpetero]

Curious your experience and guidance (or whether such questions are better asked over on the Xilinx forums, for example).

I have a few AXI peripherals.  The AXI logic to read/write from the peripherals seems to work well.  I'm not sure of the other user logic (synthesis, implementation are successful and they *seem* to work).  But that really isn't sufficient to demonstrate the user logic ALWAYS accomplishes the design intent.  I understand test benches for other RTL verilog designs and have created some.  But not sure of good practice when it comes to testing user logic inside these peripherals.  A few questions...

1. Is it good, bad, or indifferent (normal?) practice to create test benches for AXI peripherals inside the peripherals themselves?
2. Is it normal to make 'surrogate' items that represent the user logic (separate it from the AXI peripheral read/write logic) and make test benches for those items?  
3. Are there methods I'm not thinking of that are normally used for this purpose?

Just confirming it doesn't *seem* there is any issue with reading/writing from the peripheral.  I'm mostly interested in testing the user logic.

[zygot]

32 minutes ago, engrpetero said:

Is it good, bad, or indifferent (normal?) practice to create test benches for AXI peripherals inside the peripherals themselves?

I don't understand what exactly you mean here. Are you referring to adding 'instrumentation' to your source that can help debug AXI operation?

33 minutes ago, engrpetero said:

Is it normal to make 'surrogate' items that represent the user logic (separate it from the AXI peripheral read/write logic) and make test benches for those items? 

You should always write your code in a hierarchical manner so that you can separate functionality into pieces that can be understood and 'easily' verified in simulation. This is especially important for a design sources that connect to external hardware that can't be simulated as a whole. ZYNQ PS/PL connection via AXI is one example. Logic connected to an external PC host via USB is another. You can fairly easily simulate the HDL logic, but there might well be USB upstream behavior for which there is no simulation model; host software driver, application, and OS behavior. If you don't understand all of the possible behavior scenarios of the external thing that connects to your logic, it's easy to make bad assumptions that leads to bad designs. I've run into this with PC USB <--> FPGA designs. Eventually, most programmable logic designs connect to something that you can't get a model for. Sometimes, for instance memory devices, the manufacturer has a model. Often you have to make your own; and then iteratively improve the model and your HDL sources as you refine your verification.

So, that's one answer to your post. I expect that you will get other replies, including perhaps from me as I ponder your third question.

 

[engrpetero]

My term probably aren't all that accurate.  Sorry about that.

I understand the 'hierarchical manner' comment you mentioned above.  If I create 'items' (modules) that I can verify with simple test benches - and then instantiate those items within higher level items, it will be easier to understand them and their context in the higher-level items.

But I haven't figure out how to do that yet with these AXI peripherals.

For example, one of the peripherals I'm working on has 9 registers I care about.  As a skeleton peripheral - one in which I just want to be able to write to and then read from the 9 peripheral registers - I know of one easy (but relatively slow) test.  Package up the peripheral, instantiate in my top level Vivado design, connect it to the Zynq processor AXI system using other predefined IP, then write a simple C app to write and read values from the instantiated IP.  I can do that successfully - that's the easy part.

Of course, the peripheral is probably not very valuable if that's all it does.  So, I've added additional 'user logic' (I call it user logic since the Xilinx AXI templates have that term in them).

One piece of user logic I'm struggling with involves the peripheral 'acting' on different values written to one of the 9 registers (let's call them OpCodes because that's essentially what they are).  So, in the 'write' portion of the peripheral, when this one OpCode register is written to, I have a case statement to act based on the value written to that register.  It's easy to test certain portions of that case statement using the same C application.  But it is slow because I have to do the whole 'repackage, synthesis, implement, generate bit stream, update hardware specification, rebuild' process before I can test.  That isn't sustainable and there is no way that 'real' FPGA developers do that sort of testing/debug/verification.

Thinking about your 'hierarchical' suggestion, I have considered how I could create a separate module and then instantiate it as part of this design with it having an input driven from this one specific OpCode register.  But I'm not sure that is the best/correct way to do it.  My peripherals are dirt simple. so I can pivot that direction.  I'm just wondering what is sort of the 'best practice'.

 

Edited August 9, 2023 by engrpetero

[D@n]

@engrpetero,

Let's see if I can answer this.  I've now been around this bush a couple of times, and learned something new each time around, but here's what seems to work for me:

• I formally verify all bus interaction.  It's just too easy to mess that up, and end up with a design that just hangs, and at the AXI-Lite level the formal proof is pretty easy.
• Where possible, I'll formally verify the rest of the logic within each module as well.  However, my formal proofs tend to end at the leaf level.  They don't aggregate up very well into larger designs.  They're useful, therefore, just not sufficient.
• Going up a level is a bit more of a challenge, but I usually end up with a test structure similar to the picture below:
• Components include: 1) A test script.  A good test will often have many test scripts to choose from, and possibly even a perl (or other) script to test either one or all of the scripts.  2) Some kind of bus functional model (BFM).  Xilinx provides their AXI VIP for this purpose.  When using a BFM, your Verilog test script (#1) might start to read like "software"--read this register, write that register, do this computation, etc.  It all becomes quite task oriented.  3) A model of any external (typically off-FPGA) device you wish to test your design against.  I've been known to add "on-FPGA" models as well, just for testing--"device" logic that has no other purpose but to help you instrument and verify your design's interactions.  For example, I've used a GPIO module to set an "error" condition flag that I can then see in a VCD trace, or a "trace" flag that I can use to turn on and off tracing in large designs.  4) Many projects have required a RAM model of some type for my IP to interact with.  Finally, the last component of such a test structure is 5) the design under test.  This structure seems to work nicely across many of the designs I've done.
• An alternative structure I've used in the past is one where the ZipCPU acts as a CPU in this same model.  In this structure, the ZipCPU software acts as the "test script".
• This takes a bit more work to set up, however, since you now need an interconnect of some sort, address assignments, etc. in order to have something the CPU can interact with.  Were this approach not so expensive computationally (it takes a lot of work to simulate a CPU and its infrastructure, in addition to everything else), I'd use it more often--if for no other reason than you can use this approach to verify the software you'll be eventually using to interact with your device.
• This software test infrastructure is something I'd like to recommend to ARM users, but I don't think Xilinx provides a suitable PS model to make this happen.  That means, when simulating on a Zynq, you're either forced to use the BFM/AXI VIP model or a CPU that's not relevant to your application.  Perhaps there's a way around this, but I'm unaware of it.
• Regarding test "sufficiency", a test is often judged "sufficient" if it checks every logic path through your device under test.  Well, *every* is a hard target to hit.  Sometimes I end up settling with 90-95%.  Measuring how many logic paths you hit is often known as a "coverage" metric, and so IP customers often want coverage metrics for the IP they purchase.  Sadly, "coverage" feels good but ... isn't good enough.  I know that Xilinx's IP engineers are proud of their coverage metrics, yet I've still found some nasty bugs in their IP via formal verification approaches.  Still, it's a quantifiable measure, and it is better than nothing.
• As an example of an IP that supposedly has good coverage metrics, consider what happens when you attempt to both read and write from Xilinx's Quad SPI IP on the same clock cycle.  The read operation will complete with the write operations parameters.  (Or is it vice versa?  I forget ...)  This is something the AXI VIP is just not capable of testing, given how it is set up.  A good formal proof will find this--but that means you need to be doing formal verification in the first place.
• In the end, your goal is always going to be finding the bugs in your design the fastest way possible.  A good Verilator lint check can take seconds, vs a minute for Vivado to give you a single syntax error.  A good formal proof might take less than a minute, vs watching your design hang in hardware and not know what's going on.  A good simulation will go a long way towards generating confidence in your design once you do get to hardware--however, a good simulation might take longer to check than it does to build your design for hardware, put it on the hardware, and run a test on it.  At least the simulation won't require you to pull out a probe to verify things.

I've written mode about the CPU simulation approach to testing here.  Perhaps that might help answer some more of your questions.

Dan

[zygot]

I you are using Verilog or System Verilog then you have verification options that Vivado doesn't provide, like Verilator. As @D@n points out, for ZYNQ there is no clear way to "close the loop' to include the PS cores and PS/PL interconnect. Dan has published a lot of good information about AXI testing as it relates to all programmable logic implementation. I'm not sure how much of it is directly applicable to ZYNQ development flow.

For logic implemented in FPGA resources there are different kinds of simulations that can be done prior to operating the design on hardware. Behavioral simulation simply tells you if how synthesis understands your code produces behaviors that you intend. The Vivado simulator is good at finding  bad syntax or a subset of errors in your sources. It's a shame that Vivado doesn't provide good code coverage analysis that uncovers less obvious coding errors like inferred latches or incomplete case statements. In my experience most companies use tools like Synplicity for this purpose. As for whether or not your code is functionally correct, time step simulators are only as good as your testbenches.

Behavioral simulation is not sufficient because even HDL source code that is behaviorally correct and synthesized might not work in hardware after implementation. Standard time step simulation provided by FPGA logic vendor tools lets you simulate a netlist that reflects the design post implementation. This takes into account all of the delays in the signals that make it into your FPGA resources. Because the implemented logic has been changed according to your synthesis and implementation settings, many of the signals in your sources don't exist in the final implementation. But the key here is that timing closure is the final hurdle to overcome before attempting to run your design on hardware. Whether your design includes an ARM PS or and external PC or, for that matter any external device with connections to a device external to the logic resources traditional time step simulators are inadequate to include all of the behavioral or timing related scenarios that need to be verified.

Cycle based simulation is fast and certainly one of the tools that should be in your toolbox, especially for a design involving something as complex as a PS core external to the logic or even a ZipCPU implemented in logic resources. Cycle based simulation doesn't cover timing completely or, to my knowledge, even a PS core running software.

14 hours ago, engrpetero said:

One piece of user logic I'm struggling with involves the peripheral 'acting' on different values written to one of the 9 registers (let's call them OpCodes because that's essentially what they are).  So, in the 'write' portion of the peripheral, when this one OpCode register is written to, I have a case statement to act based on the value written to that register.  It's easy to test certain portions of that case statement using the same C application.  But it is slow because I have to do the whole 'repackage, synthesis, implement, generate bit stream, update hardware specification, rebuild' process before I can test.  That isn't sustainable and there is no way that 'real' FPGA developers do that sort of testing/debug/verification.

You don't have to 'package' your AXI design sources in order to do the logic part of verification for ZYNQ development. In fact I'd say that packaging your IP should only be done after simulation of your testbenches using the Vivado simulator and perhaps a third party HDL source coverage tool. Verification can't be complete at this stage because at the other end of your ZYNQ AXI bus interconnect is the PS hard block executing instructions. I have been able to cause AXI bus faults executing perfectly fine C code in the PS by doing reads or writes at too fast a rate. 

Complete verification is bridge too far for hardware or even software. The best that we can achieve is adequate verification, which as far as I know is undefinable.

AMD/Xilinx does have some documentation on verification though it is woefully inadequate, especially for the ZYNQ devices which are really ARM cores with logic rather than programmable logic with hard ARM cores from reading the documentation.

Edited August 10, 2023 by zygot

[engrpetero]

Well you guys certainly are thorough with your responses.  Much appreciated!  My delay in responding was due to having to do lots of reading about the terms and pictures you guys included.  I'm slowly getting used to Verilator - but slowly (skills to develop...).

I've gone back and done several things regarding these peripherals...

AXI bus verification

After reading (and reading and more reading), I've decided on a fairly simple but useful AXI verification.  It might not be as robust or efficient as @D@n's suggestion but I've got a limited amount of time right now with other work responsibilities.  I do hope/plan to further develop my skills and test infrastructure in the future though.

But since my current peripherals are all pretty simple - and since I've got a good, useful, template (thanks @D@n), this seems to work well. 

1. Assign initial, unique values to all registers in the peripheral.
2. Include the appropriate code to write to and read from the peripheral registers.
3. With a simple C driver for the peripheral, A) read from all registers (see that the 'initial' values are returned); B) write new values to all writable registers; C) read values from all registers and verify register values are those written to in B.

'User Logic'

1. Create appropriate Test Benches (OK to put in the peripheral itself but I've found it useful to put in a separate project so I can further simplify the Test Benches)
2. When Test Benches work as expected, (obviously) update the AXI peripheral to include the tested logic.
3. Repackage IP
4. Functionally test packaged design - this is often easy to just do with a logic analyzer or scope.

With this pretty simple approach, the Synthesis, Implementation, Generate Bit Stream, Export hardware cycle only needs to be done a few times to get something working.  I haven't really dug into the timing warnings yet.  Still some work to do there but progress nonetheless.

I'm confident I'll get more proficient and efficient, and I wouldn't have made it this far without your very useful posts so again, thanks for the responses.  In fact, I had a few 'lightbulb' moments from them that got me over the hump (well, the hump that was stopping my progress - I imagine there will be more as I develop more sophisticated items).