HLS Embedded Vision Workshop error; hls_video.h not found

[HOOKJANDRO]

Good morning,

I'm following the guide "ZYBO Z7 & Pcam 5C HLS Video Workshop" on Zybo Z7 (https://mautic.digilentinc.com/embedded-vision-workshop-download?_ga=2.155088952.1692178746.1607422380-1543877407.1597250658), however, when trying to compile edge_detect code on Vitis HLS 2020.1, the message "hls_video.h" file not found appears. I press ctrl+click on #include "hls_video.h" and it seems to open, although not legible.

Thank you.

Best regards.

[Niță Eduard]

Hi @filipj,
Good catch on the AXIS port difference
It seems to be a bug in 2020.1 where it does not get synthesized correctly:
https://github.com/Xilinx/Vitis_Libraries/issues/28
Try to use the xf_infra.hpp file from the 2020.2 version of the library and let me know if that fixes the HLS synthesis
https://github.com/Xilinx/Vitis_Libraries/blob/2020.2/vision/L1/include/common/xf_infra.hpp

Thanks,
Eduard

[JColvin]

Hi @HOOKJANDRO,

I will ask about this, though I do not know if this workshop is compatible with the 2020.1 tools; it was designed with Vivado 2017.4 and 2018.1.

Thanks,
JColvin

[Niță Eduard]

Hello @HOOKJANDRO,
hls_video.h is not found because in Vitis HLS 2020.1 it has been deprecated and replaced with Vitis Vision Library, as says here:
https://www.xilinx.com/support/answers/75345.html
Another major change in Vitis HLS 2020.1, is that it no longer provides compiled OpenCV libraries. This means that you will have to compile the libraries yourself if you want to use testbenches.

The workshop is not compatible with Vitis HLS 2020.1, but I have attached source files for a Sobel Edge Detection in Vitis HLS below (in the Vitis Vision library the Sobel algorithm  generates both X and Y gradients). I recommend that you read this post https://forums.xilinx.com/t5/High-Level-Synthesis-HLS/Using-Vitis-Vision-Libraries-and-OpenCV/td-p/1170435 since it details the process of setting up a Vitis HLS project.

If you have any more questions feel free to ask.
Have a nice day,
Eduard

edge_detection.cpp edge_detection.h edge_detection_test.cpp

[HOOKJANDRO]

Many thanks, I will try it out.

By the way @JColvin, I've had some problems here: 

 

Do you know what can be happening?

Regards.

 

[filipj]

Hello @Niță Eduard

I was trying to do exactly the same as the colleague @HOOKJANDRO. Not sure if he ever succeeded.

I was able to compile edge_detect IP from your example with vitis_hls, also was able to recompile the whole design in vivado.

However, when I run the platform, I am able to see the output from the 2 older (pre compiled, color_to_bw, invert) HLS IPs, but not from the the one freshly compiled (edge_detect). Screen remains black :(

 

I am also using Vivado 2020.1. Any hints what to to debug ? How to find out what is missing ?

Regards,

Filip

 

P.S. I have commented out #pragma HLS INTERFACE ap_ctrl_none port=return in order to get the ap_start port, but the IP does not pump output to HDMI regardless of whether it is commented out or not.

[Niță Eduard]

Hello @filipj,
Can you disconnect stream_out1_V from the AXIS switch and see if it works with one output?

Thanks,
Eduard

[filipj]

Hello @Niță Eduard,

I tried this, unfortunately without any effect :(

Tried some other experiments though, in Vitis Libraries there is a simple example that rewrites data from input to output (Vitis_Libraries/vision/L1/examples/axiconv).

I tried to make it work with Embeddedd Vision Workshop design. Unfortunately, this also had zero effect, screen remains black.

Have spotted one thing though. As I was mentioning before, color_to_bw and invert were the pre-compiled with whatever Vivado HLS the Embedded Vision Workshop was created (2018 ?) and these work just fine.

However, when I put any of the 2020.1 Vitis HLS compiled IPs, after Validate design I can see a "Critical Warning" being reported for freshly compiled ones.

It complains that TDATA_NUM_BYTES in axiconv and edge_detect is different than in the rest of the design.

I was trying to figure-out what this parameter is and how it works, without success yet...

Any further pointers ?

Thanks,

Filip

xf_axiconv_config.h xf_axiconv_accel.cpp

[filipj]

Hmm... after some more digging I've found out that the "stream" axi interface for freshly compiled IPs seems to be different than of the ones compiled with 2018.

Presuming some pragmas/settings in Vitis HLS synthesis need to be modified (or the Embedded Vision Workshop design needs to be adjusted).

 

[filipj]

Hi @Niță Eduard,

After substituting the xf_infra.hpp & recompilation of the IPs they work ! Thank you, I should have taken a look at known issues.

I guess with this we can call Embedded Vision Workshop at least initially compatible with Vitis Vision 2020.1 library, will surely make some further experiments.

One important thing to note is to remember to remove xf::cv:: from before ap_axiu, otherwise Vitis HLS will keep taking the struct definition from the wrong file.

I was able to get both edge_detection and axi passthrough IPs to work (however when I connect both outputs of the edge_detect IP it stops to work for some reason).

Thanks again,

Filip

Edited December 28, 2022 by filipj

[Niță Eduard]

Hi @filipj,
Glad you got it working.
I believe it doesn't work when connecting both output streams to the AXIS switch because you are not reading from the second one, which then causes a deadlock.
The Mat class is basically a hls::stream, which uses blocking reads and writes. https://xilinx.github.io/Vitis_Libraries/vision/2020.1/api-reference.html#xf-cv-mat-image-container-class
The sobel filter source codes reads the input stream and does both X and Y edge processing, which then gets written to two different streams. https://github.com/Xilinx/Vitis_Libraries/blob/2020.2/vision/L1/include/imgproc/xf_sobel.hpp
A simpler example would be the duplicate function, which just reads data from a stream and writes it into two different streams. https://github.com/Xilinx/Vitis_Libraries/blob/2020.2/vision/L1/include/imgproc/xf_duplicateimage.hpp

If you try to read an empty stream or write to a full stream, it will stall execution until there is data to read or space to write. https://docs.xilinx.com/r/2021.1-English/ug1399-vitis-hls/Blocking-Reads-and-Writes

In your design, the AXIS switch reads from one stream at a time. The Sobel IP writes the input to two separate streams (X and Y). The stream which is not consumed by the AXIS switch gets full, which then stalls the input stream. This causes the stream which is consumed by the AXIS to not read anymore data, which then causes a deadlock. When disconnecting one of the Sobel outputs, I think Vivado optimizes away the unused logic, so only one stream is used.

Edited December 28, 2022 by Niță Eduard
typo

[CBI]

Hi @Niță Eduard@filipj,

                    I see you guys have got the Embedded Vision Workshop working on Vivado 2020.1!! I have been trying to get it to work on my setup for some time now. These are the things that I have tried so far, but all i get is a blank screen when I pass the AXI stream through an HLS compiled block.

Steps:

1. I started by getting the raw video out by following the steps from: https://digilent.com/reference/learn/programmable-logic/tutorials/zybo-z7-pcam-5c-demo/start

So the basic video pipeline is working and the Zybo setup is good.

 

My problems start with Vitis HLS:

2. To create the edge_detect IP, I copied the necessary files from @Niță Eduardpost from Dec 10 2020. I kept the period to 6.67 (Based on the Workshop pdf.)  On compilation, I get the following timing violation.

 

 

3. I still went ahead and exported it as Vivado IP(zip file).

4. I imported the hw.xpr of the corresponding Digilent SW library and opened the block diagram in Vivado. Here I made the following connection changes:

          i) M_AXIS_MM2S -> stream_in ; stream_out1 -> video_in

          ii) ap_clk -> clk_out2

          iii) ap_rst -> peripheral_aresetn[0:0]

 

5. Exported the design XSA.

6. Imported Hardware platform into Vitis IDE, and launched on hardware.

 

Result: I only get a blank screen on my monitor. If I bypass the HLS block in my block diagram(from step 4) and repeat the whole process, I get a good feed.

 

Troubleshooting done so far:

1. Tried with axiconv as stated above, same result.

2. I tried if I could pass the feed through some already existing IP like a switch or broadcaster but no luck there either.

3. Tried changing the clock period to 10, nothing(The timing violation goes away though)

4. Initially tried with the 2020.2 suite, now on 2022.1.

5. Finally, just for the kicks, I tried booting the zybo with the pre-compiled .bin found on the tutorial, and it works just fine!! Edge detection and all!!

 

Question : Do i need to set up the OpenCV flow, if I am not planning on running simulations? My understanding is that for running directly on the hardware, the vision libraries do not need OpenCV installed.

 

It would be really great, if there was an updated tutorial as the existing ones use Vivado SDK. That said, I plan to prepare a detailed one and share it if I can get this working!!

 

Thanks!!!

[JColvin]

Hi @CBI,

I checked with Niță Eduard and learned that they still intend to look into your question and help debug it, but they did not have time to do this week, though they are hoping to get some time next week.

Thanks,
JColvin

[CBI]

@JColvin @Niță EduardThanks for the update!!

 

I tried a couple more things:

1. Edited the edge_detection.c file to remove edge detection, now it just converts the AXIvideo to xfMat and back :

#include "edge_detection.h"
void edge_detect(stream_t& stream_in, stream_t& stream_out1)//, stream_t& stream_out2)
{
	#pragma HLS INTERFACE ap_ctrl_none port=return
	#pragma HLS INTERFACE axis port=stream_in
	#pragma HLS INTERFACE axis port=stream_out1
	//#pragma HLS INTERFACE axis port=stream_out2

	//xf::cv::Mat-type local variables for intermediate results
	xf::cv::Mat<XF_8UC3, MAX_HEIGHT, MAX_WIDTH, XF_NPPC1> img0(MAX_HEIGHT, MAX_WIDTH);
	//xf::cv::Mat<XF_8UC1, MAX_HEIGHT, MAX_WIDTH, XF_NPPC1> img1(MAX_HEIGHT, MAX_WIDTH);
	//xf::cv::Mat<XF_8UC1, MAX_HEIGHT, MAX_WIDTH, XF_NPPC1> img2x(MAX_HEIGHT, MAX_WIDTH);
	//xf::cv::Mat<XF_8UC1, MAX_HEIGHT, MAX_WIDTH, XF_NPPC1> img2y(MAX_HEIGHT, MAX_WIDTH);
	//xf::cv::Mat<XF_8UC3, MAX_HEIGHT, MAX_WIDTH, XF_NPPC1> img3x(MAX_HEIGHT, MAX_WIDTH);
	//xf::cv::Mat<XF_8UC3, MAX_HEIGHT, MAX_WIDTH, XF_NPPC1> img3y(MAX_HEIGHT, MAX_WIDTH);

	#pragma HLS DATAFLOW

  //Interpret AXI-Stream interface and pull the frame from it
	xf::cv::AXIvideo2xfMat<24, XF_8UC3, MAX_HEIGHT, MAX_WIDTH, XF_NPPC1>(stream_in, img0);
  //Convert to grayscale
	//xf::cv::rgb2gray<XF_8UC3, XF_8UC1, MAX_HEIGHT, MAX_WIDTH>(img0, img1);
  //Run the Sobel operator on the x-axis with a 3x3 kernel
	//xf::cv::Sobel<XF_BORDER_CONSTANT,XF_FILTER_3X3,XF_8UC1,XF_8UC1,MAX_HEIGHT,MAX_WIDTH>(img1, img2x, img2y);
  //Convert back to RGB format for display purposes
	//xf::cv::gray2rgb<XF_8UC1, XF_8UC3, MAX_HEIGHT, MAX_WIDTH>(img2x, img3x);
	//xf::cv::gray2rgb<XF_8UC1, XF_8UC3, MAX_HEIGHT, MAX_WIDTH>(img2y, img3y);

  //Pack the frame back into AXI-Stream interface
	//xf::cv::xfMat2AXIvideo<24, XF_8UC3, MAX_HEIGHT, MAX_WIDTH, XF_NPPC1>(img3x, stream_out1);
	xf::cv::xfMat2AXIvideo<24, XF_8UC3, MAX_HEIGHT, MAX_WIDTH, XF_NPPC1>(img0, stream_out1);

}

 

2. I compiled this, the timing violation still persist (which is expected as the violation is in a datapath that does the conversion).

3. After importing the IP in Vivado, I noticed the connection wizard was prompting for auto connection. Upon clicking, it now instantiates a new block called Processor System Reset. The clk_out2 which is connected to my IP goes to this, and a reset is generated which is routed to the HLS IP. (Earlier I was connecting the peripheral_aresetn from the other processor system reset which was generated on clk_out1.)

 

 

The output is still the same though(blank screen). Now, I am pretty sure the issue is something trivial involving the interface.

4. Next I tried a run with just

stream_out1 = stream_in;

though I'm not sure if this is a legal assignment in HLS. (Same black screen)

 

Thanks!!

Appreciate your help!!

 

 

[Niță Eduard]

Hi @CBI,
The PCam demo starts with a resolution of 1980x1080p, while your HLS IP processes 1280x720p frames (based on your HLS synthesis result).
Try to change the demo resolution to 1280x720p using the UART (see Pcam 5C Image Sensor and Post Processing Options) or modify the starting resolution (change this line pipeline_mode_change(vdma_driver, cam, vid, Resolution::R1920_1080_60_PP, OV5640_cfg::mode_t::MODE_1080P_1920_1080_30fps); ).
Let me know if this helps.

Thanks,
Eduard

[CBI]

Hi @Niță Eduard,

              Yes that was it!! Thanks a ton!! Should have paid more attention to the video resolutions!!

 

For folks who are starting out on this, I have created a step by step tutorial. https://gitlab.ssec.wisc.edu/cbabu/pcam_edge_detection

 

Regards,

Charls

Edited April 18, 2023 by CBI
To add documentation