Hello.

And welcome to a new tutorial series

on The Coding Train about a piece of software

called Runway.

So what is Runway?

How do you download and install Runway and kind of tinker

around with it?

That's all I'm going to do in this particular video.

Now, let m be clear, Runway is not something that I've made.

Runway is made by a company, a new company

called Runway itself.

And it's a piece of software.

You can use it and download it for free.

You can use it for free.

There are aspects of it that require Cloud GPU credits,

which I'll get into later.

And you can get some free credits and a coupon code

that you'll find in the description of this video.

But really I want to just talk to you

about what it is cause I'm so excited about it,

and I'm planning to use it in the future,

in a lot of future tutorials and coding challenges, and teaching

things that I'm going to do.

And I also should just mention that I

am an advisor to the company Runway itself.

So I'm involved in that capacity.

All right.

So what is Runway?

Right here it says machine learning for creatives.

Bring the power of artificial intelligence

to your creative projects with an intuitive and simple

visual interface.

Start exploring new ways of creating today.

So this, to me, is the core of Runway.

I am somebody who's a creative coder.

I'm working with processing and P5JS.

You might be working with other pieces of software.

That's just commercial software, coding environments.

You're writing your own software.

And you want to make use of recent advances

in machine learning.

You've read about this model.

You saw this YouTube video about this model.

Can you use it in your thing?

Well, before Runway one of the things you might have done

is find your way to some GitHub repo that

had like this very long ReadMe about all

the different dependencies you need to install and configure.

And then you've got to download this and install this, and then

build this library.

And you can really get stuck there for a long time.

So Runway is an all in one piece of software

with an interface that basically will run machine learning

models for you, install and configure them

without you having to do any other work

but press a button called Install.

And it gives you an interface to play with those models,

experiment with those models, and then broadcast

the results of those models to some other piece of software.

And there's a variety of ways you

can do that broadcasting, through HTTP requests,

through OSC messages.

And all these things might not make sense

to you, which is totally fine.

I am going to poke through them and show you

how they work, with an eye towards at least showing you

how to pair Runway with processing,

and how to pair Runway with P5JS,

and I'll also show you where there's lots of other examples

and things you can do with other platforms, and stuff like that.

So the first step you should do is click here

under Download Runway Beta.

It will automatically trigger a download

for Mac OS, Windows, or Linux.

I've actually already downloaded and installed Runway.

So I'm going to kind of skip that step,

and just actually now run the software.

Ah.

And now it's saying, welcome to Runway.

Sign in to get started.

OK.

So if you already have an account,

you could just sign in with your account.

I do already have an account.

But I'm going to create a new one, just so we can

follow along with the process.

So I'm going to go here.

Create an account.

I'm going to enter my email address, which is-- shh.

Don't tell anyone-- daniel@thecodingtrain.com.

Then I'm going to make a username and password.

Now that I've put in my very strong password,

I'm going to click Next.

And I'm going to give my details, Daniel Schiffman,

The Coding Train.

Create account.

Ah.

And it's giving me a verification code

to daniel@thecodingtrain.com.

Account has now been created, and I can click Start.

So once you've downloaded, installed Runway, and signed up

for an account, logged into your account,

you will find this screen.

So if you've been using Runway for a while,

you might then end up here, clicking on open workspaces,

because workspaces are a way of collecting

a bunch of different models that you

want to use for a particular project into a workspace.

But we haven't done any of that.

So the first thing that I'm going to do

is just click on Browse Models.

So the first thing that I might suggest that you do

is just click on a model and see what

you can do to play with it in the Runway interface itself,

because one of the things that's really wonderful about Runway

is as a piece of software and an interface you can explore

and experiment with the model to understand how it works,

what it does well, what it doesn't do well,

what it does at all, before starting

to bring it into your own software or your own project.

So I'm going to pick this Spade Coco model, which I have never

looked at before.

This is very legitimate me.

I have no idea what's going to happen when I click on that.

And now, here I can find out some more information

about the model.

So I could find out what does the model do?

It generates realistic images from sketches and doodles.

I can find out more information about the model.

For example, this is the paper that describes this model,

"Semantic Image Synthesis with Spatially Adaptive

Normalizations Trained on COCO-Stuff Data Set."

Remember when someone asked, is this a tutorial for beginners.

Well, it is for beginners in that you're a beginner.

You can come here and play around with it.

But you can go very deep too if you want to find the paper,

read through the notes, and understand

more about this model, how it was built,

what data it was trained on, which is always

a very important question to ask whenever you're

using a machine learning model.

So we can see there are attributions here.

So this is the organization that trained the model.

These are the authors of the paper.

We can see the size of it, when it was created,

if it's CPU and GPU supported.

We could also go under Gallery.

And we can see just some images that have been created.

So we can get an idea.

This is a model that's themed around something

called image segmentation.

So I have an image over here.

What does it mean to do image segmentation?

Well, this image is segmented, divided into a bunch

of different segments.

Those segments are noted by color.

So there's a purple segment, a pink segment,

a light green segment.

And those colors are tied to labels in the model,

essentially, that know about a kind of thing

that it could draw in that area.

So you could do image segmentation in two ways.

I could take an existing image, like an image of me,

and try to say, oh, I'm going to segment it.

This is where my head is.

This is where my hand is.

This is where my hand is.

Or I could generate images by sort

of drawing on a blank image, saying put a hand over here.

Put a head over here.

So that's what image segmentation

is, at least in the way that I understand it.

What have I done so far?

I've downloaded Runway.

I've poked around the models.

And I've just clicked on one.

Now, I want to use that model.

I want to play with it.

I want to see it run.

So I'm going to go here to Add to Workspace.

It's right up here.

Add to Workspace.

Now, I don't have a workspace yet.

So I need to make one.

And I'm going to call this workspace,

I'm going to say Coding Train Live Stream.

So I'm going to do that.

I'm going to hit Create.

Now, I have a workspace.

You can see, this is my workspace.

I have only one model added to this workspace over here.

And it's kind of highlighting up for me right now what to do.

I need to choose an input source.

So every machine learning model is different.

Some of them expect text input.

Some of them expect image input.

Some of them might expect input that's

arbitrary scientific data from a spreadsheet.

Then the model is going to take that input in, run it

through the model, and produce an output.

And that output might be numbers.

Or it also might be an image.

Or it might be more text.

So now we're in sort of the space of a case by case basis.

But if I understand image segmentation correctly,

I'm pretty sure the input and the output

are both going to be an image.

Let's make a little diagram.

So we have this--

what was this model called again?

Spade Coco.

So we have this machine learning model.

Presumably there's some neural network architecture in here.

Maybe it has some convolutional layers.

This is something we would want to read that paper

to find out more.

Runway is going to allow us to just use it out of the box.

And I certainly would always recommend

reading more about this to learn more about how to use it.

So my assumption here is in my software that I want to build,

I want to maybe create a drawing piece of software

that allows a user to segment down an image.

So you can imagine maybe I'm going to kind of draw

something that's one color.

Look, I could use different colored markers.

I'm going to sort of fill this image in with a bunch

of different colors.

And then I am going to feed that into the model.

And out will come an image.

So we have input.

And we have output.

And again, this is going to be different for every model

that we might pick in Runway.

Although, there's a lot of conventions.

A lot of the models expect images

as input and output images.

Some of them expect text as input, and output an image,

or image as input and output text.

Et cetera.

And so on and so forth.

And so now what I want to do is choose the input source

in Runway for the model.

So something that's going to produce a segmented image.

So that could be coming from a file.

It could actually come from a network connection, which

I'll get into maybe in a future video,

or you can explore on your own.