Hello and welcome to another video tutorial

about working with Runway and running machine learning models

in Runway itself.

Now, before you watch this video tutorial,

if you've never used Runway before,

you might want to go back and look at my Introduction

to Runway, how to download and install it.

But to be honest, you probably can figure that out

if you just go to runwayml.com and click on Download Beta.

You're going to want to download and open the runway software.

You'll go then to Browse Models.

I might go here under Motion, and I'm going to click PoseNet.

And you'll find yourself right here.

So this is where I am.

I've installed Runway.

I've downloaded it.

And I'm on the page in the Runway software for the PoseNet

machine learning model.

Now, what is PoseNet?

PoseNet is a machine learning model

that performs real-time skeletal tracking of one or more people.

And guess what?

I'm a person, and I've got Runway running here

with PoseNet.

So I'm going to run it.

So let's actually first click Add To Workspace.

So I already have a workspace that I've

made in the previous video called Coding Train Live

Stream.

I want to choose an input source, which

I want it to be my webcam.

So, yep, Runway go right ahead, and there I am.

And then I want to choose an output source, which eventually

I want to be processing, because I want

to get the results of running this machine learning model

PoseNet into processing itself.

But for right now, I'm just going to click on Preview.

So I click on Preview.

Oh, and I have to run.

But guess what?

So this is different than what I've showed in previous videos.

I've got an option for Run Locally.

And, in fact, this model can only be run locally.

It would be silly to run this one in the cloud,

because I'd have to spend all this time sending

the data over the network.

And it's very easy for it to run.

This is a very small, fast model.

It can be run on most modern computers.

So I'm going to click Run Locally.

So it requires no GPU credits.

Absolutely, can be used for free.

And we can see there it goes.

It's running right now.

It is making guesses as to where the various key points

of my skeleton are on my body in the output That's. viewable

below.

So one of the nice things about working

with Runway and its models is a lot

of times models have different parameters and values

and things that you can tweak and change to try running it

in different ways.

And these are sort of known as hyper parameters

to a machine learning model.

And so some of them I would actually

have to stop running the model and then

I can start to play with it.

So, for example, this Architecture one

is something I can actually make the models smaller.

It might be less accurate, but it will run faster.

But so, for example, I'm just going

to change this to 0.75 instead.

I'm going to run it again.

But some of these parameters can actually

be tweaked in real time.

So, for example, I can change the width and height

of the image, which is actually changing

the resolution of the image from the webcam itself.

And I can make it more grayscale if I want.

I could do various things to actually tweak

the image before it goes in.

But this is not the important piece

of what I want to do in this video.

What I want to do in this video is

we have a moment here where I've got a model running in Runway.

And I'm able to play with it, tweak it, get it exactly

the way I want it to work.

And I want to take that next step

from having it run here to be able to see the result of it

in my own piece of software.

So let's make that happen.

This software that I'm going to use to attempt this

is something called Processing.

So since this size here in Runway of the output

is 640 by 362, what I'm going to do in my Processing code

is set the size of the canvas to 640 by 362.

void draw.

background 0.

So now, I have a Processing sketch,

which I am running right here.

How do I see the results, the output of the model

in my Processing sketch?

So there are a variety of different network protocols

that Runway supports.

And I can find out about them up here

by clicking this Network tab.

And the one that I want to use for working with Processing

is OSC.

So there's a variety of reasons why you might

pick one protocol over another.

It really depends on what you're doing.

In the case of where I just want to get a single image,

an HTTP request would make the most sense.

And I'll do that in another video

when I show you how to work with style again in Runway.

But right now, I'm going to click

on OSC, which works pretty well with Processing.

And it's telling me a lot of information here.

So it's saying, hey, this is the server address.

So this is the most important thing that I need from Runway.

It's what I'm going to tell Processing is the unique IP

address, which happens to be the local IP

address of this computer and the port number

from which it can get the OSC messages.

I'm going to click here.

And I'm going to create a string called like ip.

I'm pretty sure I'm probably going

to want the port number in a separate variable.

And I'm going to create a port number like this.

Now, I could sit here and write all the code

for this, which is what I usually

do in Coding Train videos.

But this is a fairly different circumstance.

I really just want to get the example up

and running and show you how to do this.

And one of the nice things about working with Runway

is there are a whole bunch of premade examples

for you with different platforms and pieces of software,

one of which is Processing.

So let me show you how you would actually

do this in the real world, how I would be doing this, which

is the way to do it right now.

So if I go to the Runway ML GitHub--

and I actually should go back one level

and go here under Runway ML.

You can see here's the GitHub page for the Runway software.

And there's a lot of information.

There's some sort of like high-level stuff

here about how to port your own machine learning model

to Runway itself.

So if you've trained your own model or you

find a model that's not supported by Runway,

how you could add it.

But that's not what we're really doing here.

What I want to look for is here Processing, Runway

and Processing.

If I click here, this repository has

a whole bunch of examples of using Runway with Processing.

So you can see there's a StreetView one, attnGAN,

face landmarks, im2txt, and, voila, PoseNet.

This is the one that I'm working with.

So I'm going to look at this example.

I should say that this is an open source project.

Processing is an open source project

that I'm involved with that I've talked

about in a lot of other videos.

So maybe this is a place where if you find another model

in Runway that you've made work and you want to contribute

your Processing example here, I would encourage you to do that.

And Chris, one of the founders and creators of Runway,

and I have been talking about making a Processing

library for Runway.

And it just so happens that I made two recent video tutorials

about how to make a Processing Java library.

So I see a project in the future, which is a Processing

library for Runway.

So if you want to get involved with that, with making that,

write in the comments and let me know.

And maybe we'll create a GitHub repo for that.

OK, so I mean to go here under PoseNet.

And I'm just going to click here under posenet.pde.

So I could just copy paste the whole thing,

but I'm going to kind of go piece by piece.

And you can see here already Runway host and Runway port.

So I made a mistake in my code.

So I got a-- oh, I'm missing the 0.1 here,

which I'm sure the Chat is already talking about.

And there's an extra 1 here.

And this should be an integer.

And this would really be Host.

So actually let me use the same variable

names, runwayHost and runwayPort.

OK, so I think I've gotten this right now.

Now, in order for the example to work,

I need to make sure also I have the Processing OSC

library installed.

So if I come over here, I can copy

paste these import statements.

I can copy paste this OSC object that I definitely need.

And we'll see very quickly that I

have an error, which is the class OscP5 does not exist.

This is because I haven't installed the Processing OSC

library.

You might have already have it installed because you use it

with a different project.

But I could go here.

And I could do Sketch, Import Library,

Add Library, OSC to search for it.

This is the library I'm looking for oscP5.

I'm going to click Install.

And now, you'll see that error message goes away.

The next thing that I want to do is create an object

to receive OSC messages.

And I think I can just make this a new OscP5 object.

I need to give it a reference to this particular sketch,

because it's going to need to trigger events in the sketch

when there's data available.

And then just give it the port number.

This is called runwayPort.

So if you look at the runway example,

it's actually using this object called OscProperties, which

is a bit more sophisticated.

It says a remote address, a listening port, a datagram

size, and some other stuff.

I should probably just copy paste this into my example,

but I'm curious if it will work with just the sort of more

simplified default OscP5 object, where I just say

this and the particular port.

But I do need this.

I do need a broadcast location, because I'm

going to have to send messages to Runway as well, saying I'm

connected or I'm disconnected.

So there's two things at play here.

There's the Processing software.

And there's the Runway software.

They're both running locally on my computer.

Now, it's possible that in other scenarios

there could also be a cloud GPU involved

that runway sends messages back and forth between.

And this is something that I'm going

to do in the next example, where I work

with something called StyleGAN.

I'm going to have Runway also talk to a cloud GPU.

But that's not happening here.

PoseNet is actually running inside of--

basically wrapped into Runway itself locally.

So PoseNets running here in Runway locally.