Welcome to part 3 of the tutorial. Over at Beetlebox we are excited for the release of Vitis, which is the unification of all of Xilinx’s previous software into a single package. We have been working hard on computer vision using this platform and thought that we could provide some help to others wanting to get started on Xilinx’s development boards. This tutorial will be a multi-part series covering the basics of getting started with computer vision and Vitis and will be covering:

• Getting XRT and Linux up and running
• How to build “Hello World” and understanding Vitis Development Flow
• Using OpenCV on the embedded system (Current Page)
• Using Vitis Vision Library
• Accelerating Computer Vision using XRT and Kernels

We hope these tutorials will be useful for anyone looking to get into computer vision on FPGAs.

Part 3: Using OpenCV on the Embedded System

OpenCV

OpenCV is one of the most popular Computer Vision libraries in the world and forms the backbone of many projects, including on FPGAs. This tutorial we will be focused on running OpenCV on our ARM core, giving us a solid foundation for accelerating our computer vision in later tutorials.

OpenCV comes pre-installed on our Petalinux system, but we still need to do a bit of fiddling with our compiler to be able to use it, so in this tutorial we will be setting it up and running through a simple example of making an image greyscale.

Pre-installed version of OpenCV

For 2019.2, the pre-installed verison of OpenCV is 3.4.3 comes with following enabled:

• python3
• eigen
• jpeg
• png
• tiff
• v4l
• libv4l
• gstreamer
• samples
• tbb
• gphoto2

It even comes with the experimental versions of OpenCV. Most notably missing from this installation version is ffmpeg which may be needed to read certain video files. We won’t worry about this yet, however, and instead just focus on images.

Transfering files in Software Emulation and in Hardware on Xilinx FPGAs

Using computer vision requires us to test on images and videos, hence knowing how to transfer files properly is critical. This tutorial will also cover transferring images on and off our Zynq.

Instructions

Pre-requisites:

• Follow the previous tutorial guide to have a basic working system

Our particular setup:

• Operating System: Ubuntu 18.04.02
• Development board tested: ZCU104

Getting the OpenCV libraries in place

• Begin from the system project we created last tutorial;
• File->New->Application Project
• In the ‘Create a New Application Project’ window:
  • Project Name: opencv
  • Click ‘Next’
• In the ‘Platform’ window:
  • Click the platform that we created in the previous tutorial
  • Click ‘Next’
• In the ‘Domain’ window:
  • Click ‘Next’
• In the ‘Templates’ window:
  • Click ‘Vector Addition’
  • Click ‘Finish’
• In the Explorer window:
  • Under ‘opencv’ right click the ‘src’ folder
  • Delete ‘krnl_vadd.cpp’ and ‘vadd.h’
  • Go to ‘opencv.prj’ and remove ‘krnl_vadd’ from the Hardware Functions
  • Rename ‘vadd.cpp’ to ‘main.cpp’
• In ‘main.cpp’ replace the code with the following:

#A simple greyscale colour convert in OpenCV

#include <opencv2/core/core.hpp>
#include <opencv2/imgcodecs/imgcodecs.hpp>
#include <opencv2/imgproc/imgproc.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/features2d/features2d.hpp>
#include <opencv2/xfeatures2d.hpp>
#include <opencv2/videoio/videoio.hpp>

#include <iostream>
#include <stdio.h>
using namespace std;
using namespace cv;

int main()
{
	Mat src_image;
	Mat grey_image;

	src_image=imread("test_image.jpg");

	if (!src_image.data)
	{
		cout << "Could not open image" << endl;
		return 0;
	};

	cvtColor(src_image, grey_image, CV_BGR2GRAY);

	imwrite("grey.jpg", grey_image);

	cout << "Created grey image" << endl;

    return 0;
}
 

• Make sure to begin with that we are in software emulation mode
• When we save the file, we will notice a lot of red under our includes, so we need to configure our setting to solve that
• By default the OpenCV that Vitis links is the one that is contained in Vivado. This is problematic for us because it isn’t the exact same installation as we have generated with Petalinux, meaning we will get version clashing
• The way we can fix this is by going into the installation directory of Vivado, which be default is

/tools/Xilinx/Vivado/2019.2/include

• Remove the ‘opencv’ and ‘opencv2’ folders (We can just cut the folders up a level in case we ever need to restore it).
• We now need to tell the Vitis where the correct folders are

In the Explorer window:

• Right click: ‘opencv’-> ‘C/C++ Build Settings’
• The window ‘Properties for opencv’ should appear
• Under ‘GCC Host Compiler’ click ‘Includes’
• Under the ‘Include paths’ window, add a new line:

<xsa_build_path>/xsa_build/zcu104_base/src/aarch64-xilinx-linux/usr/include

• Under the ‘GCC Host Linker’ click ‘Libraries’
• Under ‘Libraries (-l)’ add the following libraries:
  • opencv_core
  • opencv_calib3d
  • opencv_imgcodecs
  • opencv_imgproc
  • opencv_highgui
  • opencv_features2d
  • opencv_xfeatures2d
• Under ‘Library search path (-L), add a new line:

<xsa_build_path>/xsa_build/zcu104_base/src/aarch64-xilinx-linux/usr/lib

Emulating our program in software

• To test our system we first need an image, called “test_image.jpg”

• We need to upload this image to our QEMU so we can run opencv on an actual image
• Copy and paste ‘test_image.jpg’ into the ‘src’ folder
• In the ‘Explorer’ window, click ‘Emulation-SW’->’_vimage’->’emulation’->’sd_card.manifest’
• In ‘sd_card.manifest’ add the following line:

 <workspace_directory>/opencv/src/test_image.jpg

• Run the simulation, by right clicking ‘opencv’ in the ‘Explorer’ window and clicking ‘Run As’->’2 Launch on Emulator (Single Application Debug)’
• Wait for the program to finish at which point the console should display:

Created grey image

• In the ‘Emulation Console’ window type

ls

• ‘grey.jpg’ should be listed in the results. We now need to extract that data by setting our link between our host and VM
• Click ‘Xilinx’->’XCST Console’
• This should open a XCST Console window which we can use to transfer files. Open the link to the VM by typing in the console:

connect -url tcp:localhost:1440
ls

• Our host directory should be listed we can now transfer files using two commands:

tfile copy -from-host /path/host /path/vm
tfile copy -to-host /path/vm /path/host

• Let’s transfer our grey image off our QEMU:

tfile copy -to-host /mnt/grey.jpg  <host_image_directory>/grey.jpg

• Looking at our image directory we should have a grey image

• As a final note, it would have been possible for us to transfer the image on to the system using XCST and then execute the program, but transfering the file on boot saves a little bit of time

Running our program on the Zynq itself

• Since there is no kernels to emulate, we will skip the hardware emulation part and run our program straight on hardware
• Swap configurations by going to the ‘opencv.prj’ file and changing ‘Active Build Configuration’ to Hardware
• The C/C++ build settings are not saved between different build configurations, so we need to repeat the process of getting the OpenCV libraries in place
• Once all the libraries are back in place, clean the project by right clicking on ‘opencv’ and clicking ‘Clean Project’
• Build the project
• The project build will be quick here because there are no kernels meaning Vitis does not need to perform any synthesis. The issue is that Vitis will only output ‘opencv.exe’ in our ‘sd_card’ folder. Luckily, we can just re-use the SD card image from our previous tutorial and transfer our .exe file on
• Boot and connect the Zynq platform as in previous tutorials
• To transfer our files we will be using scp.
• Connect an ethernet cable from the Zynq board to the host
• We need to configure our Zynq’s address by running the following commands on the Zynq:

 ifconfig eth0 inet 192.168.1.10 
 ifconfig eth0 netmask 255.255.255.0 

• On the host, the connection needs to be configured using the Internet Protocol Version 4 (TCP/IPv4) and setting the properties to:

IP address: 192.168.1.11
Subnet mask: 255.255.255.0 

• How to do this will vary depending on the operating system, but once it is set, we should be able to ping our board from the host:

ping 192.168.1.10

• The board should respond. Now we can copy over the files using the following commands on the host:

scp  <host_directory>/test_image.jpg root@192.168.1.10:/mnt
scp  <vitis_directory>/opencv/Hardware/sd_card/opencv.exe root@192.168.1.10:/mnt 

• When prompted for the password, use ‘root’
• On the Zynq in ‘/mnt’ , we can run ‘ls’ to see that ‘test_image.jpg’ and ‘opencv.exe’ have been placed on the board, run the executable:

./opencv.exe

• Using ‘ls’ again the new image ‘grey.jpg’ should have appeared. We can transfer that data out with the scp command on the host:

 scp root@192.168.1.10:/mnt/grey.jpg   <host_directory>

• Opening up jpeg we get our flower:

Now we have a basic framework in place for getting image data into our FPGA, processing it and then getting it back out. Using this we now have the basis for accelerating our computer vision. Next time we will look into what Vitis Vision Libraries and how we can use this great resource for accelerating Computer Vision.

Previous: Building a “Hello World”

Next: Vitis Vision Library

If you have enjoyed this tutorial but are in current need of talent to build advanced Computer Vision systems on FPGAs, consider joining our ClickCV Early Access programme. ClickCV Early Access provides bespoke service and support for developing advanced Computer Vision systems on FPGAs. We use our own proprietary Computer Vision Library, ClickCV, to provide our clients the cutting edge in low latency, high definition processing. Contact us today to find out how we could build your next-generation system.

About the Author: Andrew Swirski is the founder and managing director of Beetlebox, a Computer Vision Acceleration specialist. The company develops the Computer Vision Acceleration library called ClickCV, which is designed to fully utilise the hardware adaptability and performance of FPGA chips. Beetlebox is currently running an Early Access programme, where the company provides bespoke service and support to develop client’s Computer Vision systems on FPGAs. Before Beetlebox, Andrew Swirski previously worked at Intel (formerly Altera) in FPGA encoding and decoding. He completed a Masters’ in Electrical and Electronic Engineering from Imperial College London in 2017.