Filters and the nice thing about separable filters is that they're incredibly quick

And this means that for example in computer games you could actually do some of this on the graphics card

At 60 frames a second, right? These are the kind of things that people who are writing computer games

Not me people who know what they're doing

well, I think if you the games are thinking about right, how can we do post processing like motion blur and

Bloom and other kinds of filters that look good in a game, but don't drive the whole thing to a halt

Separable filters are a great way of doing this

The problem is that Gaussian blur which is the thing

We kind of focused on

Doesn't look that good as a blur right if you say well I want to do some depth of field

I want to do some optical depth of field

All right

So I want the foreground person in my game to be nice and sharp and I want the background to be lovely blur, right?

It will look a bit odd. If you use a Gaussian because that's not what a lens does

You need to use a proper bokeh blur. I don't know how you pronounce that where bouquet is right bouquet right bokeh bouquet

Oh, yeah, I lost interest. It's because of the way that the optics of a camera work, right? So a

Gaussian your point is spread over a sort of Gaussian function. So it decreases like this, right?

That's not what happens in an optical blur my lens blur

So if we remember back a while ago, we did a light-filled video

You've got a point light source here, which is something in your scene a person a light. They've tree not important, right?

It's gonna be quite a few of these if they if your image has more than one pixel in it

You have a lens right which looks like this

it might have other glass optical elements and you have an imaging plane here like this and

What happens is point light don't just travel directly into your camera. Everything just emits light in lots of directions depending on its properties

so it comes in like this and when it's in focus

It will come and it will hit a point on here and that pixel will obtain that color

the color will be the sum of all of these light rays and then it sort of simplified to be usually between nought and

255 if you're out of focus

What will happen is these light rays will come pass the camera like this and focus on a point behind it

And actually you'll have this light will now be spread over this whole area

From the front if this is the front of our square sensor every point will now be a circle like this

and each other point will also be a circle and another circle in another circle and this gets you that kind of

Characteristic Lens Blur that you see when something like light sources christmas tree lights or something are out of focus

Cut to a nice shot of this so people spend a lot of money on lenses

Primarily, I think because of the quality of this kind of blur. They're not perfect circles all the time

Sometimes there's slightly odd all their hexagons because you've got an aperture and things like this

They give this blurry sort of unique characteristic even down to a lens which you can't really get on a computer

But we can have a go because we've got to do it at 60 frames a second

You haven't got the option of thinking about Zeiss and like although this is most obvious when you're looking at point light actually this applies

To any blur right? If you do a Gaussian blow of a scene, that doesn't have very bright bits

It will still look a bit different than if you have a normal bouquet blur

Right, and and actually you look much better. If you do it properly. Can we recreate our

Separable filters, but now perform a lens blur or at least an approximation to a lens blur

And the answer is not it's not so easy, but you can kind of have a go

You have to start looking at imaginary numbers a little bit

But if we just sort of not worry about it too much you essentially replace your separable

Convolutions with two complex separable convolutions and we can start to start to get something going

So one particular game engine that does this is EA's Frostbite engine which drives a lot of its sports games

So things like Madden will have a nice blur on certain shots that's created by an artificial lens blur

Not a Gaussian blur other games might kind of fake it a bit of a Gaussian

I don't know Primavera folks. It's not technically correct. There's a kind of short paper on this

There's a few blogs online that talk about stepper lens blur. There's some implementations out there

I've got my own implementation that I wrote which I'll release at the end of this video

There was a nice presentation at the Game Developers Conference in 2017 showing office technology, right? So, this is real-time

Bouquet blur not Gaussian blur right? So how does it work?

well

the idea is that what we want to do is instead of having our kernel which is sort of like this and a

Gaussian we need to have it be a circle right in an ideal world that right

Which from the top looks like a nice cut clear-cut circle now you can do that

But you can't separate it

you can have a

kernel

Which is jazz a load of zeros and then a circle of ones in the middle and that will produce you a very nice filter

But it will take ages because you have to do all this point wise and it's not separable

So what we need to do is find a function that we can split into a vertical pass and a horizontal pass

All right, let's look at example, right one of the nice things about Gaussian is but it's separable right? It's a naturally separable function. So

How can we convert a Gaussian to something that kind of approximates this circular shape?

And the answer is that we can sort of add in an imaginary component and create sort of complex Gaussian

Now obviously we're not gonna go into huge amount of detail on complex numbers

But they have an imaginary and a real component. The imaginary component is multiplied by I which is a square root of minus 1

the good news for us is that most of the time when we actually need to use it the eyes get multiplied out and they'd

a sort disappear or become a minus number

All right

That's all buddy time gonna go into actually it's a bit like when we talked about our discrete cosine

Transform and when maybe some time we'll talk about Fourier transforms

We have waves that add together to create something the shape we want in this case

The wave is going to be a Gaussian, right? But it's going to go up and down like this

So instead of our Gaussian going like this, which is not quite centered what we're going to roll on it

We have one that goes up and down right in the complex the complex

Component of this Gaussian is the bit that makes it do that

So we have a Gaussian that sort of goes down like this and sort of centers around here and it comes down like this

All right. It's symmetrical

And then we have another one but perhaps come sort of like this

and when you add them together you

get the thing we want so these kind of cancel out and you sort of get some of looks a bit like this with a

Bit of wobbling around and then it comes up about here and then kind of goes like that and then down like that

Right and that is our circle and that is also separable because it's essentially got a Gaussian as it's sort of primary function

So I've coded this up and I've been using it to mess around with different blurs

Most of the code is to do with creating the kernels

I do some normalization and things like this then I pass the image from each component and the very end

I combine them Mayon. I'll draw a little diagram to show how that works

the more components you use a bit like in

JPEG or any kind of sort of adding of waves the more ways you use the better approximation. You're going to get right?

So if you're not watching it for a game engine, you can use maybe two each time you do this

You've got quite a few convolutions you can have to do these are complex multiplication operations in addition

It takes time, even on a graphics card, right and bear in mind. There's other game things to do, right? The game isn't static

with just this nice blur

So you've got other stuff to be worrying about fact that there's an imaginary number involved in some sense to me

Anyway, personally, I like to look at it. It's basically not making any difference at all

The only difference is but the addition and multiplication that you would normally do in a convolution is now a complex addition and a complex

Multiplication which are sort of well defined structures that you can just apply and this is a mathematical term not just as saying it's different

It's absolutely a mathematical term right and it's actually not very complicated. You have your image, right?

We have our first component C naught here and we have another component so you want we could have lots of errs

this has a

Complex kernel so that we'll have a real and imaginary right but they're just going to look a bit like gaussians like I showed you

This is going to be the same over here. These are different ones

And the idea is that when we add these two together, we get a very nice disk

so what we're going to do is we're going to

converge with these and we're going to converse in which with these and we're going to add them together at the end because I see

A lot of addition goes on. All right

so we take our image we go through here and we get another image out, which is our

Real and another image out which is our imaginary we go through here we get an image out. That's real and image out

that's an imaginary and

Then we add these two together

in a weighted sum and then we take our fine or weighted images for each component and we add them together in our

Final output like that. The reason we have to do these separately is so that we can make it separable, right?

This is this can be done separately

This can be done in the same way very very quickly, right if you're doing it for just two components

There's a few convolutions you have to do add it all together and you have your result the SEPA bility means that instead of doing

all the kernel for every location

we do a vertical pass of a strip and we take that output and put it through a horizontal pass and

Then we combine the outputs at the end and that's that's it mathematically equivalent to doing the 2d version but much much faster

So let's have a look at some outputs right now

I've written some code here very straightforward that essentially produces these complex kernels in one dimension and kind of higher than 10 image

You know in the correct order add them up at the end so we can blur smooches, you know

I'll release the code so you can have a look the original post that led me to this work

Outlined a few parameters for these kernels that butt looks good, right they heat optimized so I put them in as well

So let's have a look a couple of these kernels and see what they are. So if I just look at the zero

The one component pass, right?

So this is if you want to try and create a disc with just one of these complex kernels, like what would it look like?

Well, let's have a look. These are the real and imaginary parts of this particular component

And this is going to look roughly like a disc when we add these two together. Let's see. So if I take the two together

We can run it and it's kind of a doughnut he disc right?

And the reason is because you can't get a perfect disc with just a real and imaginary part added together

It's not going to work. So what we can do is we can take another kernel add that to it

and now we've got kind of two wheels and two imaginaries in some loose sense add these all together and maybe we can start to

Approximate this disc so we'll get a little bit better if I change this

And run that together

All right. So now it's got a bit of a ripple to it

But this is starting to look quite a lot better. Like this looks to me quite a lot like the disc

I'm hoping for and this is still separable so I can take a strip from the middle with a strip here and

I can run it in two passes and save a huge amount of time

because this is a sort of a this is I think a

64 or 65 by 65 pixel kernel, this will be a big deal if you wanted to run this over an image

Especially, you know a sort of resolution of modern games run out so we can get even better we can add more and more components

So if I've ramped up the number of components like we get somebody looks really really quite good. There we go

so this now looks I think quite a lot like a disc so essentially

This is equivalent to running a circle of ones in a convolution over an image

It's just that now we can do it in these separable components and save ourselves a huge amount of time. It's not perfect, right?

The size is difficult to manage. The edge is not as sharp as I'd like this should be a sharp edge

But you can't get an absolutely sharp edge, but for a sort of game engine something in the background, it's gonna look great

And it's going to run really really quick in the EA and in the Frostbite engine

They'll only use two components so it won't be a perfect

Colonel but it will look pretty good stars are a classic example of some where we can use this blur

Because of course their point lights have a nice spread out nicely

So this is what happens if we Gaussian blur this image, it looks dull and light just like not high-resolution and doesn't look great

right, if we if we look at the lens version, I mean

I think it's aesthetically better, right?

When it looks like this, right each of these points has spread out into a nice disc

I've had to do some exposure here because it's that's difficult to do but I think it looks pretty good

So that's the Gaussian and that is the lens blur. I think that looks a lot better

so this is our fake lens blur by using our

Our circular disc kernel with lots of components because I wasn't worried about speed right?