Original subtitles

In this coding activity we are going to write the code for objects detection on image.

We have input image, configuration file that describes parameters for every layer, trained weights and names

for COCO

classes. We will use all these with OpenCV deep learning library and will get detected objects on resulted

image. It is expected that you have basic knowledge on how YOLO version 3 works.

But anyway, you can refresh your knowledge and find brief description of algorithm in PDF attached

to this lecture. You will also find short description of used parameters in configuration file. And other

useful links.

Even if you don't know anything about YOLO, step-by-step with a lot of practice in this course you will

get principal idea. You can pause video now, read PDF for some time and come back whenever you are

ready. Okay, let's jump to the code now.

This is file with name yolo-3-image.py and we will use it to detect objects on given input

image. Algorithm is as follows:

Firstly, we read input image and get from it

so-called blob.

Then we load YOLO version 3 network and implement forward pass with blob. After that, we get bounding

boxes and filter them with technique called Non-maximum suppression.

Finally, we draw found bounding boxes and labels on original image. As a result, we will have OpenCV window

with original image and bounding boxes with labels around detected objects.

Let's go through the code.

We import numpy and OpenCV libraries.

Also, we will use library to measure time spent for forward pass. In this block

we read input image by OpenCV function that give us BGR image in form of numpy array. We prepare

OpenCV window giving the name and specifying that it is re-sizable with argument WINDOW NORMAL.

To show original image

we use function imshow() in which we pass

name of the window and image itself.

Also, we have here checkpoints, let's uncomment them.

This checkpoint shows us shape of input

image.

We prepare variables for height and width of input image that we are going to use later in the code.

And below there is checkpoint that shows these height and width.

Let's comment everything else and have a look at our input image.

Just select all lines of the code below and press Control Slash.

Let's run the code.

Here is our input image

and we can see shape of the numpy array, height and width. Okay,

let's continue. Let’s uncomment

next block.

In this block we prepare so called blob from image that gives us preprocessed image that we will feed to

the network. To get the blob

we use OpenCV function blobFromImage() that takes image itself in form of numpy array, scale factor

by the help of which we normalize image dividing every element of array to 255, desired size

of output blob from image and we specify that we don't need image to be cropped. And one more argument

about swapping Red and Blue channels,

we set it to True, because we opened input image by OpenCV library that gave us image in BGR order

of channels. This function returns 4-dimensional tensor which is blob and shape of which we check

by this checkpoint. Let's uncomment it.

We will see that blob has shape with number of images, which is one, number of channels and size of

the image. We also can show blob in OpenCV window with next checkpoint.

But firstly, we need to slice the blob to get only needed numpy array of image, and move channels to

the end with transpose() method where we specify needed order. Let's uncomment this checkpoint and have

a look at the blob.

Let's run the code.

Here is our input image, and here is our blob from input image.

Good,

let's continue. Let’s uncomment

next block.

Now we load YOLO version 3 network.

We read file with names of COCO classes and put them into the list that we are going to use when drawing

bounding boxes around detected objects. With this checkpoint

we can see all 80 classes in the list.

Let's uncomment it.

Now we load our trained YOLO v3 network by OpenCV deep learning library. We use function

readNetFromDarknet(), specifying path to configuration file, and to trained weights. After we load our YOLO

version 3 network we need to get only output layers' names: yolo 82, yolo 94, and yolo 106, because we are

going to use them later to get response from feed forward pass.

That's why, we firstly get all layers' names with function getLayerNames() and we can use checkpoint to

print all layers' names inside YOLO version 3. Let's uncomment

it.

And then, we get only output layers' names with function getUnconnectedOutLayers() and put them into

the list.

With this checkpoint we can see resulted list. Let's uncomment it.

Also, in this block we set minimum probability to threshold with in order to eliminate all weak predictions.

And we set threshold to filter weak bounding boxes by non-maximum suppression technique. Later, when

you use this code for your own purposes you can tweak these two parameters to find better performance.

And finally, for this block we generate colours for bounding boxes.

We use numpy function for generating integer numbers, specifying low and high boundaries, size which is 80

classes with three numbers for Red, Green and Blue channels.

By this checkpoint we can see the shape of generated numpy array and have a look at numbers of colour

for the first class.

Let's uncomment this checkpoint and have a look.

Let's run the code.

Here is our input image,

here is our blob from input image.

Here are our names for classes. Here

are our all layers' names. Here are our output layers' names. And here are our generated numbers of colour for the first

class.

Great,

let's continue.

Let’s uncomment

next block.

We have prepared everything for implementing forward pass.

Let's do it.

We use function setInput() for setting as input

our blob. And we use function forward() for getting output results only for specified layers that we prepared

earlier.

Also, we measure time spent for forward pass with time library specifying start point and end point.

Here we print resulted time in seconds.

Let's have a look

but firstly let's comment back

all previous checkpoints.

Okay,

let's run the code.

Here is our input image. And here is time spent for forward pass to detect objects. Yours time should be

different and depends on machine you use.

Great, let's continue. Let’s uncomment

next block.

After we get response from output layers we can collect all detected objects and corresponding bounding

boxes. Firstly, we prepare list for collecting bounding boxes,

list for corresponding confidences and list for class numbers. Then, we go through output layers and through

all detections in every layer.

In the first for loop we iterate output layers and in the second for loop we iterate every detection.

Every detected objects variable is given in form of numpy array where first four numbers represent

coordinates of bounding box that is normalized to real width and height of original image, and rest 80

numbers represent probabilities for every class that this bounding box might belongs to. In variable

scores we collect all 80 probabilities for current detected object. Then, in variable class current we

write index of class with highest probability and, in variable confidence current we write value of highest

probability for current detected object.

Then, we eliminate weak predictions by checking if current confidence is higher than minimum probability

that we set earlier.

If it's higher, then we get first four numbers from detected objects where first two are centre point

of detected object and last two are width and height. These four numbers are normalized by width and height

of original image,

that's why we can just multiply them elementwise in order to scale up bounding box and fit it to original

image size. We multiply elementwise sliced numpy array with four numbers to another numpy array

also with four numbers. As a result, we get centre point of detected object in original big image and

its real width and height. In order to use OpenCV windows to draw bounding box we need to calculate

top left point. We have centre point, width and height. And we calculate top left corner by subtracting

half of width and half of height.

Finally, we add current detected object to the prepared lists: bounding box top left corner, width and height,

maximum confidence, and class number that is one out of 80. Good, let's continue. Let’s uncomment

next block.

When we eliminate weak predictions by minimum probability we get all bounding boxes with confidences

above this number. But it can happen that some of bounding boxes overlap each other. Which one to choose

then? In order to answer this question, we use so called non-maximum suppression technique that filter

non-needed bounding boxes if their corresponding confidences are low or there is another bounding box

for this region with higher confidence. We use function NMSBoxes() and pass as arguments bounding

boxes, their confidences, minimum probability and threshold that we defined earlier.

As a result, in results variable

we get final bounding boxes.

Great,

let's continue.

Let’s uncomment

final block.

We've got needed bounding boxes

and now we can draw them and label them.

We check if there is any object being left after filtering by non-maximum suppression and go through

indexes. These indexes

we use to access the needed bounding boxes, corresponding class numbers and confidences in the lists

that we defined and filled earlier, before non-maximum suppression.

Also, we define counter for detected objects,

and here we print the class number for detected object.

Now we get coordinates of current bounding box and its width and height.

Then we prepare colour for current class that we generated in form of numpy array earlier and convert

it into the list with method tolist() in order to use it with OpenCV function to draw bounding box.

With this checkpoint

we can check that it was converted from numpy array to list type.

Finally, we can draw bounding box with function rectangle() where we pass original image, coordinate of

left top corner, coordinate of right bottom corner that we calculate by adding to left top corner width and

height of bounding box. And we pass colour for current class, and thickness of the line for bounding box.

Here we prepare text to show above bounding box with label and confidence. We start to draw text a little

bit above from left top corner of bounding box by function putText() where we also pass original image,

prepared text, starting point, font style, font size, colour which is the same with colour of current bounding

box, and thickness. And we also print number of detected objects before and after non-maximum suppression.

Okay, we are ready to show final results in the window with original image and bounding boxes with labels

on it. Let's do it. Let's run the code. Here is our input image, here is resulted image. Here is time

spent for forward pass to detect objects. List of objects with their labels. And comparison results of

total objects been detected before non-maximum suppression and left after filtering

with this technique.

In the end of the file you can find section with useful comments. Well done! We detected objects on

this image. How much time did it take for forward pass in your computer? Share your results in Question

and Answer board!

Let's now move to activity where you will detect objects on given image.