Zybo Z7 HMDI Demo - the starting point for a video manipulation project

[Boneoh]

Hi to all!

I'm new to FPGA and I'm motivated to learn as much as I can about it. I saw the Zybo Z7 HDMI demo project, I think its awesome!  I just ordered the Zybo Z7 -20 board. I've downloaded the project source and have been reading through it. I realize I've got a lot to learn but I'm patient, persistent, and have time to work on it. 

My goals are

1. Learn this project as best I can to understand the details.

2. Make small changes over time to implement some manipulations to the video. I'm thinking about glitchy processing for video art. Simple things like bit shifting of the image data, quantization of pixel values, etc. 

3. In the longer term I'd like to implement some manipulation functions based on the luma values, things like increasing or decreasing the luma of the video buffer or filtering base on luma, etc.

I already have a few questions.

1. The HDMI Demo seems to never use index 0 of the frame buffer array. This is frameBuf in video_demo.c,  is that used for some special purpose? 

2. My video workflow is all 1080p30, the vga_modes.h file has an entry for 1080p60. My knowledge is very limited, but it seems like I should be able to just change the .freq entry, the remaining parameters for porch etc. are the some. Am I correct?

3. My background is strong in software development, limited in hardware and electronics. How should I approach adding manipulation to the project?  Should I be learning about DSP slices? Approaching it on the software side? 

Boy, I've got a lot to learn! Thanks in advance for any pointers.

Pete 

[Eran Zeavi]

Hi, in vga_modes.h you can find a list of all supported video formats.
I have the same issue as you, I Iam software and embedded software developer, but my knowledge with video is limited.
I am right now looking to downscale the frame rate from 60Hz down to 5 Hz t and still trying to figure it out.
What is your project about ?
 

[Boneoh]

Hi, Eran! I want to learn about HDMI in FPGAs so I can use the frame buffers to manipulate the video in real time. This will be for glitchy video art.

I have a Zybo Z7-20 and am trying to load it up. It took two days on my old PC to setup Vitas + Vivado and update to 2024-1-1. I need to replace the hard drive with an SSD !

It seems that the file "Zybo-Z7-20-HDMI-sw.ide.zip"  for 2024-1-1 is missing the  "Zybo-Z7-20.src" folder. This is in the -10 version of the zip file, the folder contains the source code. I will create a new post specific to this, I'm guessing that the src folder is the same for -20 as the -10 version, but I really don't have a clue.

Missing .src folder?

Edited August 19 by Boneoh
Added link to new topic.

[Boneoh]

I used to 'src' folder from the -10 version and it worked great. The HDMI Demo is up and running.

Quote

I am right now looking to downscale the frame rate from 60Hz down to 5 Hz t and still trying to figure it out.

I think you can add a new entry by editing  the source file that has the video settings, it is file "vga_modes.h" in the "src\display_ctrl" folder. I am trying something similar in a couple days, I want 1080p30. I should be able to add a new entry and change the .freq setting. Then in file "src\video_demo.c" there is function "DemoCRMenu" that will need to be changed to add the new mode to the prompt. Last, in function "DemoChangeRes" it will need to be changed to use the new mode when the user enters the response to the prompt.

Here is a link  that shows the parameters for many video modes. 

[Boneoh]

I've been making progress getting up the learning curve. I've been able to add the LEDs, buttons, and switches to the HDMI Demo project. 

My challenge is how to generate an interrupt from the PL to the PS to handle the framebuffer. I'd like to have the software in PS receive an interrupt to know when to switch framebuffers, so the software will be working in one buffer while the PL is displaying another. 

I've tried experimenting with the code, but I'm too new at this to get a good understanding of what to do. Here is what I've found.

A. The v_tc_in is generating an interrupt when receiving the lock. This works great. The v_tc_in.irq is connected to xlconcat0.In3 and the xlconcat0.dout is connected to processing_system7_0.IRQ_F2P[4:0]      I'm guessing that these are mapped when the function fsInitInterruptController in intc.c is called.  But it's not clear to me what is happening there. The five bits coming out of the xlcontact0.dout are going to bit 4:0 of the PS.  I seem to be missing the critical point here...

B. I see the code in video_demo.c where the "ivt" is setup for both GPIO and VTC. The interrupt id is set to HDMI_IN_VTC_IRPT_ID which was set to XPAR_FABRIC_V_TC_IN_IRQ_INTR. 

C. Code in video_capture.c function named VtcIsr receives the interrupt. 

Getting more specific here:

1. Should I be using an interrupt from v_tc_in to detect a new frame and switch buffers? It seems that v_tc_out is used for the output timing. The only use I can see for the v_tc_in is simply for the 'locked' interrupt. 

2. What interrupt is best to use? The product guide shows an example video on page 54, Figure 3-8. Here the "fsync" values are used by the video pipeline. 

3. What do I need to do to implement this?  a) add a new entry to the ivt. b) add a connection for one or more of the fsync bits to the xlconcat. c) create a handler to receive the interrupt.

Thanks in advance!

Pete 

[Eran Zeavi]

Hi,

There is a new recent issue (about a week ago) with the HDMI demo for zybo-20, after months it has been working for me.
I reinstalled fresh Vivado and HDMI demo , not solving the issue.
Vivado 2024.1 fails synthetising the project due to "ila_pixclk module not found" error showed on the picture.
My guess is that the ILA IP upgrade killed it but not sure 

Any idea ?

Thanks

Eran
 

Edited September 5 by Eran Zeavi

[Boneoh]

@Eran Zeavi 

I am running Vivado + Vitas 2024.1.1 but seem to be getting the same messages. 

I am so new to FPGAs that I don't understand the implications of these messages. The demo software does seem to be working for me.

Are you able to run the demo at all? Does anything appear to be broken? Is there anything you would like me to try on my machine? 

I get these critical warning messages in Vivado when generating the synthesis and implementation. 

Edited September 5 by Boneoh
Replaced image

[Boneoh]

I just found this and will dig deeper. https://github.com/Digilent/vivado-library/issues/25

 

[Boneoh]

I've been struggling to try to get the PL to PS interrupts added to the HDMI demo. I've added the getting started with baremetal code into the HDMI demo and can turn the LEDs on and off using the buttons and switches.  The goal is to have the VTC in the demo generate an interrupt to the PS. This would use a Frame Sync out from the VTC into the concat and then the IRQ_F2P of the PS. I've struggled with this for a few days and decided to try a more limited test case.

I've changed the Getting Started With Baremetal example to try to test the interrupt with no success. If someone would take a look, I would really appreciate it. 

Thanks!

Pete

GettingStartedBaremetal.ide.zip

[artvvb]

I haven't had much of a chance to look at the HDMI project itself, apologies. The warnings about the ILA should at least be ignorable, as Eran Zeavi pointed out. I've also let the folks who did the last round of demo updates know about the missing .src folder.

1 hour ago, Boneoh said:

I've changed the Getting Started With Baremetal example to try to test the interrupt with no success. If someone would take a look, I would really appreciate it. 

There are some missing API calls - you should be enabling the AXI GPIO interrupts through the xgpio.h header, at a minimum, some calls to XGpio_InterruptEnable and XGpio_GlobalInterruptEnable are missing. Once they're working, the interrupts need to be manually cleared in the handler by the processor as well. If you go poking around in the driver headers and c files, there's a lot of good information. This modified version of your main file works on my machine:

#include "xparameters.h"
#include "xil_printf.h"
#include "xgpio.h"
#include "xil_types.h"
#include "xscugic.h"

// Get device IDs from xparameters.h
#define BTN_ID XPAR_AXI_GPIO_BUTTONS_DEVICE_ID
#define LED_ID XPAR_AXI_GPIO_LED_DEVICE_ID
#define SWT_ID XPAR_AXI_GPIO_SWITCHES_DEVICE_ID

#define BTN_CHANNEL 1
#define LED_CHANNEL 1
#define SWT_CHANNEL 1

#define BTN_MASK 0b1111
#define LED_MASK 0b1111
#define SWT_MASK 0b1111

XScuGic InterruptController;
static XScuGic_Config *GicConfig;

u32 btn_data;

void ExtIrq_Handler(XGpio *InstancePtr)
{
	u32 irq_sts;
	irq_sts = XGpio_InterruptGetStatus(InstancePtr);
    XGpio_InterruptClear(InstancePtr, irq_sts);
	btn_data = XGpio_DiscreteRead(InstancePtr, 1);
    xil_printf("ExtIrq_Handler: %08x\t", irq_sts);
    xil_printf("%08x\r\n", btn_data);
}

int SetUpInterruptSystem(XScuGic *XScuGicInstancePtr)
{
  Xil_ExceptionRegisterHandler(
		  XIL_EXCEPTION_ID_INT,
		  (Xil_ExceptionHandler) XScuGic_InterruptHandler,
		  XScuGicInstancePtr
  );
  Xil_ExceptionEnable();
  return XST_SUCCESS;
}


int interrupt_init(XGpio *InstancePtr)
{
  int Status;

  GicConfig = XScuGic_LookupConfig(XPAR_SCUGIC_0_DEVICE_ID);
  if (NULL == GicConfig) {
    xil_printf("interrupt_init error - XScuGic_LookupConfig\r\n");
    return XST_FAILURE;
  }

  Status = XScuGic_CfgInitialize(&InterruptController, GicConfig, GicConfig->CpuBaseAddress);
  if (Status != XST_SUCCESS) {
	xil_printf("interrupt_init error - XScuGic_CfgInitialize\r\n");
	return XST_FAILURE;
  }

  Status = SetUpInterruptSystem(&InterruptController);
  if (Status != XST_SUCCESS) {
	xil_printf("interrupt_init error - SetUpInterruptSystem\r\n");
	return XST_FAILURE;
  }

  Status = XScuGic_Connect(
		  &InterruptController,
		  XPAR_FABRIC_AXI_GPIO_BUTTONS_IP2INTC_IRPT_INTR,
		  (Xil_ExceptionHandler)ExtIrq_Handler,
		  (void *)InstancePtr
  );
  if (Status != XST_SUCCESS) {
	xil_printf("interrupt_init error - XScuGic_Connect\r\n");
	return XST_FAILURE;
  }

  XScuGic_Enable(
		  &InterruptController,
		  XPAR_FABRIC_AXI_GPIO_BUTTONS_IP2INTC_IRPT_INTR
  );

  return XST_SUCCESS;
}

//void XGpio_InterruptGlobalEnable(XGpio *InstancePtr);
//void XGpio_InterruptGlobalDisable(XGpio *InstancePtr);
//void XGpio_InterruptEnable(XGpio *InstancePtr, u32 Mask);
//void XGpio_InterruptDisable(XGpio *InstancePtr, u32 Mask);
//void XGpio_InterruptClear(XGpio *InstancePtr, u32 Mask);
//u32 XGpio_InterruptGetEnabled(XGpio *InstancePtr);
//u32 XGpio_InterruptGetStatus(XGpio *InstancePtr);

int main() {
	XGpio_Config *cfg_ptr;
	XGpio led_device, btn_device, swt_device;
	u32 data;

	xil_printf("Entered function main\r\n");


	// -------------------------
	// 'Normal' processing
	// -------------------------

	// Initialize LED Device
	cfg_ptr = XGpio_LookupConfig(LED_ID);
	XGpio_CfgInitialize(&led_device, cfg_ptr, cfg_ptr->BaseAddress);

	// Initialize Button Device
	cfg_ptr = XGpio_LookupConfig(BTN_ID);
	XGpio_CfgInitialize(&btn_device, cfg_ptr, cfg_ptr->BaseAddress);

	// Initialize Switch Device
	cfg_ptr = XGpio_LookupConfig(SWT_ID);
	XGpio_CfgInitialize(&swt_device, cfg_ptr, cfg_ptr->BaseAddress);

	// Set Button Tristate
	XGpio_SetDataDirection(&btn_device, BTN_CHANNEL, BTN_MASK);

	// Set Button Tristate
	XGpio_SetDataDirection(&swt_device, SWT_CHANNEL, SWT_MASK);

	// Set Led Tristate
	XGpio_SetDataDirection(&led_device, LED_CHANNEL, 0);

	XGpio_InterruptEnable(&btn_device, 0b1111);
	XGpio_InterruptGlobalEnable(&btn_device);

	// -------------------------
	// Interrupt processing
	// -------------------------

    interrupt_init(&btn_device);

	// -----------------
	// Loop forever
	// -----------------

	while (1) {
		data = btn_data;
		data &= BTN_MASK;
		if (data != 0)
		{
			data &= LED_MASK;
		}
		else
		{
			data = XGpio_DiscreteRead(&swt_device, SWT_CHANNEL);
			data &= SWT_MASK;
			if (data != 0)
			{
				data &= LED_MASK;
			}
			else
			{
				data = 0;
			}
		}

		XGpio_DiscreteWrite(&led_device, LED_CHANNEL, data);
	}
}

Serial prints, ignore the top part, just the bottom five lines represent booting the board, pressing and releasing BTN3, then pressing and releasing BTN0.

[Boneoh]

@artvvb Thank you so much for the quick and detailed reply! I will try this asap and confirm the results. 

Regards,

Pete

[Boneoh]

@artvvb Yay! This works great!

Thanks a  bunch for getting me over this hurdle!

Edit: I've added the switches to the interrupts following your example. I've attached a zip file for the ide.

Regards,

Pete

GettingStartedBaremetal.ide.zip

Edited September 7 by Boneoh
Added .zip file

[Eran Zeavi]

@Boneoh The demo was running just fine for months. Since 3 weeks ago, I am experiencing recurrent and random  crashes of Vivado during "generate block design" and I have synthesis generating a blank "synthesized  Design". Reinstalling Vivado-2024 for Windows from scratch did not help. Reverting last Windows updates did not help; reinstalling JAVA did not help. I have systematic Vivado crashing or failing synthesis.
Not sure how to go from there ...
Are you running Vivado on Windows or Linux ?

Thanks
 

[Boneoh]

@Eran Zeavi I am running Vivado and Vitas 2024.1.1 on Windows 11.  The Critical Warnings that I am getting don't result in the "Synthesis Failed" hard stop that you are getting.  These are the critical warnings I am getting when using the original HDMI Demo source project.

 

[artvvb]

@Boneoh

More of a heads up, there was some more discussion of this issue in this other thread last Friday:

 

[Boneoh]

@artvvb thank you for the heads-up!  I have tried a help out a bit, but I am so new for FGPAs that I'm afraid that I'm not much help. I'm following the thread and am happy to test on my setup, but so far I've not had any roadblocks. 

The good news for my little project is that I've been able to use the HDMI demo to learn and have some fun. 

Progress:

1. Learn the tools - I'm starting to understand the basics.

2. Added LEDs, buttons, and switches to the block diagram.

3. Creating my first IP block to start manipulation of the video pipeline.

I feel like the small wins are keeping me motivated!

[artvvb]

On 8/1/2024 at 11:07 AM, Boneoh said:

1. The HDMI Demo seems to never use index 0 of the frame buffer array. This is frameBuf in video_demo.c,  is that used for some special purpose? 

Jumping aways back. I can't find the commit where it was updated, so I'm going off of memory and one old archived email, but this was a change that got introduced a couple years ago - it was an easy workaround for a bug in the two command line options that modify (invert or scale) the incoming framebuffer and then display the result on the output, which had problems only when the display buffer that was being "printed" into was at buffer index 0. Specifically, the output buffer would be temporarily stored, and then be overwritten when the input started back up (virtually immediately - the debugger was needed to catch it), even if the input was pointed at a different buffer. We didn't find a cleaner/better fix for it at the time than to effectively use indexing that starts at one for the videocapture and display_ctrl drivers.

Edited September 16 by artvvb
cleaned up last sentence

[Boneoh]

8 minutes ago, artvvb said:

Jumping aways back. I can't find the commit where it was updated, so I'm going off of memory and one old archived email, but this was a change that got introduced a couple years ago - it was an easy workaround for a bug in the two command line options that modify (invert or scale) the incoming framebuffer and then display the result on the output, which had problems only when the display buffer that was being "printed" into was at buffer index 0. Specifically, the output buffer would be temporarily stored, and then be overwritten when the input started back up (virtually immediately - the debugger was needed to catch it), even if the input was pointed at a different buffer. We didn't find a cleaner/better fix for it at the time than to effectively used indexing that starts at one for the videocapture driver.

That's interesting! I made changes to the code to allow display of buffer with index set to zero and just saw a blank display. One of the hardest parts for me learning FPGAs is understanding the 'why' of some things. I'm doing OK learning the 'how' with the documentation, but sometimes I feel a bit lost.

[thinkthinkthink]

Release archive for 20/HDMI should be fixed now.

@Eran Zeavi Try to shorten project file paths as much as possible and make sure you don't have empty spaces in the paths like for example "c:\hdmi project". Rename folder to "c:\hdmi_project" or "c:\hdmi-project".

@Boneoh Make sure Stack and Heap size are at least 0x2000 (or 8 KB) in the linker script lscript.ld.

[Boneoh]

I'm learning and making progress on this project!  

I've added a simple custom IP block to shift the video data bits with wraparound. Also using interrupts for buttons and switches. This is fun! 

 Video with bit shifting

The next step is to use BRAM to control the IP processing. I'm using the GettingStartedWithBaremetal example to test my changes, then implementing the changes to my modified HDMI example. I was able to implement a BRAM and access it through the PS. The BRAM will be written by the PS to a set of small structures that contain the control parameters. The structures are small, less than 100 bytes each. There will be 6 to 10 structure instances mapped into the BRAM. The data will not be changed often and latency is not an issue.  

I am lost as to how to implement the BRAM access on the PL side. I've attached an image of the block design. How should I approach reading the BRAM in the PL? I was thinking that I should create a custom IP that accesses the BRAM and allows the data to be read. 

Thanks in advance!

Pete 

 

Edit: typos and replaced block design png with better image.

Edited September 22 by Boneoh

[Boneoh]

I found this post  that shows how to use an AXI BRAM Controller. It looks like what I am missing, I will give it a try.

[artvvb]

Looks like a good resource. A dual-port BRAM where the AXI interface controls one port and the other hardware in PL controls the other port is the way to go. It'd also be possible to implement this as a custom AXI4-Lite IP or RTL module, but the documentation on how to approach either of those isn't the clearest.

[Boneoh]

I'm having trouble trying to get an FSYNC interrupt from the PL to the PS. I see in file video_capture.c function Gpio_Isr where the interrupt for the timing lock is setup. I've added the callback for the frame sync and the bit for fsync all to the interrupt enable call. The callback function does not receive the interrupt. 

		// $$$ This is where the interrupt handler is set for the timing Lock

		XVtc_SetCallBack(&(videoPtr->vtc), XVTC_HANDLER_LOCK, VtcIsr, videoPtr);

		// $$$ Try similar for FSync

		XVtc_SetFSync(&(videoPtr->vtc), 1, 0, 0);

		XVtc_SetCallBack(&(videoPtr->vtc), XVTC_HANDLER_FRAMESYNC, VtcIsr_F_SYNC, videoPtr);

		XVtc_IntrEnable(&(videoPtr->vtc), XVTC_IXR_LO_MASK | XVTC_IXR_FSYNCALL_MASK );	// $$$ Added OR for FSYNC_ALL

 I've read through the VTC documentation but I must be missing something.  I'm also confused as to which VTC is actually generating the lock interrupt, but that's not a problem other than my lack of understanding. This is in function VideoInitialize.

	/*
	 * Setup direction registers, and ensure HPD is low
	 */
	XGpio_DiscreteWrite(&videoPtr->gpio, 1, 0);
	XGpio_SetDataDirection(&videoPtr->gpio, 1, 0); //Set HPD channel as output
	XGpio_SetDataDirection(&videoPtr->gpio, 2, 1); //Set Locked channel as input $$$ ? VTC_OUT ?

Here is the relevant part of the block design for vtc_in

And vtc_out

[artvvb]

5 hours ago, Boneoh said:

I've read through the VTC documentation but I must be missing something.  I'm also confused as to which VTC is actually generating the lock interrupt, but that's not a problem other than my lack of understanding. This is in function VideoInitialize.

The device ID for the input VTC is passed into the video_capture driver via the call to VideoInitialize in the video_demo.c source. It's the one named "v_tc_in", as the xparameters definition tries to match the IP names in the block design.

Can't help much with the fsync interrupt issue, at least today. You could potentially check whether individual fsync pins are going high at the expected times with an ILA.

[Boneoh]

Thanks for the feedback. The past hour or two I've been reading through the Vivado Programming and Debugging Tutorial. I really need to learn the debugging part of the process flow now. It's number one on my list.

It seems I need to be looking at how to do this with Vitas, since the demo has a lot of dependencies on the software side. Also the app takes about 15 seconds to load and sync up the video. 

Any ideas of a guide or tutorial on how to do this in Vitas Classic Mode?  The HDMI demo docs say that the Classic is required.

I am able to get through the process of "Synthesis, Set Up Debug" ok. But I think that I am only half way there.

Thanks a bunch for all your help!

Pete