All language subtitles for 2. 3D Projection

In the previous video we started to clean up the Retimer

artifacts in this clip of the skateboarders.

One of the techniques I showed you used a 2D transform to track

In a patch to fix the front of the truck.

In this video I'm going to show you how we can use a 3D

setup to help us track into more patches.

To use this technique you'll need a 3D camera track for your shot.

You can use the Camera Tracker node in Nuke X to

solve a 3D camera from your footage.

Track your original footage, not your retime footage.

We can apply the retime to the camera track afterwards.

Guessing a good camera track is a detailed process,

which I'm not going to cover in this tutorial.

In this case, you can see that I have a decent solve,

and if we view the point cloud that the camera tracker

generated you can clearly see the details for the ground and

the walls on either side of the street.

We're going to use this camera to project patches in place

to clean up some of the retime issues.

The projection technique works by reversing the purpose of the camera,

and using it to project an image onto a piece of geometry in the 3D scene.

Projection is different from just applying an image directly to a

piece of geometry because the position of the image on the surface

of the geometry is based on the camera,

not on how the UVs are set up for the geometry.

This means that as long as the geometry is in the correct position in 3D

space your image ends up in the correct part of the surface based on

where it was in the 2D image to start with.

Usually you use a static frame from your footage and freeze the

animation of the camera to the same frame.

You then project the frame and rerender it using the

animated version of the camera.

The result is that your static frame sticks to the correct 3D

position as the animated camera moves relative to it.

Camera projection is a really useful technique for painting out

things in your scene that you don't want to be there.

In this case, we're going to use it to paint out the retime artifacts.

Let's set up an example projection before we worry about the retime.

We're going to work with the original speed footage

for this part of the demonstration.

Let's change the color of the manhole cover and use a

projection to track in the effect.

I'm going to use frame 227 as the reference frame.

Let's connect a Roto node to the Read node, and draw around the manhole cover.

Then add a pre-multiply node below the roto to isolate it.

Now let's add a FrameHold node below that,

and set it to 227 to freeze the frame.

Let's add a Grade node above the roto and make that image hot pink.

Now we have a single frame of the new cover.

We need to track this back into the moving image.

Let's look at the point cloud from the camera track.

As part of the sole process I orientated the 3D scene,

so that the points that correspond with the road line up the

ground plane in Nuke's 3D environment, but it's not quite perfect.

We need to create a car to represent the surface of the road.

If we connect our viewer to the Camera Tracker node,

and open it up,

and set the Display option to Points we can see the tracked

features that it used to solve the scene.

We can select some of the points on the road and right click and choose create,

card.

This will give us a card that lines up through the

average of all the selected features.

If we look in 3D space we can see the card amongst the selected points.

We're going to project the pink manhole cover onto this card.

Let's move the card under the FrameHold node.

If we connect this directly we would apply the cover to the

card as a texture based on the UV coordinates,

but this isn't what we want.

We need to project the image onto the card with a Project3D node,

but before we do that we need to remove the LensDistortion from the image.

Any time we take a 2D image into a 3D setup we should remove the

LensDistortion first and then reapply it afterwards.

Copy the LensDistortion node that was created with the CameraSolve,

and connect it below the FrameHold.

Make sure to set it to Undistort.

Now we can connect a Project3D node.

This node needs two inputs,

the image to project and the camera to project it with.

We need to project the image with a static camera that matches frame 227.

We can copy the FrameHold node,

and place it between the Camera and the Project3D node.

Now we can connect the Project3D node to the card.

If we look in the 3D space we can see the result.

Now we can connect the card to a ScanlineRender node.

We want to render this out with the animated camera,

so connect it directly to the Camera node without the FrameHold.

Then we need to add another LensDistort node,

this time with Undistort unchecked.

Then merge this over the original image.

When we play the result you can see that the new pink cover

tracks nicely over the original image.

At the moment,

this is working on the original speed footage because the

camera track was done before the retime.

If we want to apply this to the slower footage we need to

render the projection through a version of the camera with

the retime applied to it.

We can do that with an expression similar to one we used in lesson three.

I've imported the script from the previous video.

This has the TimeWarp node that we were using as the

master to control the Kronos nodes.

We're now going to connect our render camera to the same master control.

Let's copy the camera.

In the new camera I click on the graph icon next to the

translation properties and choose Edit expression.

We need to add a set of brackets after the word curve, and then add TimeWarp1.

lookup inside them.

Copy and paste the expression into the x, y, and z values and close the dialog.

Paste the same expression into the rotation values too.

You'll need to do this for any other animated properties in your camera.

In this case, the focal length has animation key frames,

but the value doesn't change over time.

We could add the expression, just to be safe.

Now I have a camera with a retime that matches our slower footage.

In the Node tab of the camera add RETIMED to the labels,

so that we can tell the difference in our script.

Because we are using the original footage to set up this patch we still need

to use the original camera to project it onto the card,

but if we render it with a new retime camera,

and merge it over the retime footage, you can see that it lines up.

We're going to use this technique of projecting patches to do a much better

job of cleaning up the road at the lower edge of the frame.

I'm going to work backwards from the last frame,

which is frame 245 of the original footage.

Let's create a frame hold for that frame of the non-retime footage.

I'm going to create a roto shape and select as much of the road

as I can without including any of the skaters,

and I'm going to add a little bit of a feather to the shape.

Pre-multiply the image by the roto,

and then add a copy of the LensDistortion set to undistort.

Now we can project this patch onto a copy of the same

card that we used for the manhole cover.

Use the Project3D node and connect it to the original unretime camera.

Add another FrameHold node between the camera and the Project3D,

and set it to 245, then connect it to the card.

We might need to increase the overall scale of the

card to fit the whole patch onto it,

but don't change the position of the rotation.

Then connect to ScanlineRender to the card and hook that

up to the retime version of the camera.

Because we are using the retime version of the camera we

might not see anything on frame 245, so let's move to frame 280.

Now reapply the lens distortion, and merge this over the retime setup.

When we play this back we can see that the patch sticks well to the road,

but there's a change in brightness as the patch arrives into the shot.

This is due to the vignette around the edge of the lens.

We could try and fix this with some animated grades and mattes,

but we're going to need to create some other patches

to fix the rest of the shot anyway.

I ended up repeating the set up for every third frame of the original footage,

and merging them with the earliest frames over the later frames.

I then created an overall mask to keep these

patches from going over the skaters.

I've also held out the DissolveWatermark,

which is obviously going to cause some issues.

When we play this back we get a pretty smooth result,

and it also solves the vignette issues around the edge of the frame.

We can see a bit of an edge to the patch.

We probably need a softer edge to our roto,

but then we would start to overlap the scatter in his shadows.

This is all stuff we can fix given more time.

Now that we have this retime camera setting we can create cards

throughout the scene to track in more patches.

If we placed a vertical card,

placing the camera at the correct depth for the front of the black truck,

we could have used this approach to track in the patch that we

created to fix the truck in the previous video.

As I mentioned at the beginning of this module,

we could spend days cleaning up everything in this shop,

but with time, and the three techniques we've covered in this module,

you should be able to get a decent result.