Placeholder Image

字幕表 動画を再生する

  • [DING]

  • Hello.

  • And welcome to a new tutorial series

  • on The Coding Train about a piece of software

  • called Runway.

  • So what is Runway?

  • How do you download and install Runway and kind of tinker

  • around with it?

  • That's all I'm going to do in this particular video.

  • Now, let m be clear, Runway is not something that I've made.

  • Runway is made by a company, a new company

  • called Runway itself.

  • And it's a piece of software.

  • You can use it and download it for free.

  • You can use it for free.

  • There are aspects of it that require Cloud GPU credits,

  • which I'll get into later.

  • And you can get some free credits and a coupon code

  • that you'll find in the description of this video.

  • But really I want to just talk to you

  • about what it is cause I'm so excited about it,

  • and I'm planning to use it in the future,

  • in a lot of future tutorials and coding challenges, and teaching

  • things that I'm going to do.

  • And I also should just mention that I

  • am an advisor to the company Runway itself.

  • So I'm involved in that capacity.

  • All right.

  • So what is Runway?

  • Right here it says machine learning for creatives.

  • Bring the power of artificial intelligence

  • to your creative projects with an intuitive and simple

  • visual interface.

  • Start exploring new ways of creating today.

  • So this, to me, is the core of Runway.

  • I am somebody who's a creative coder.

  • I'm working with processing and P5JS.

  • You might be working with other pieces of software.

  • That's just commercial software, coding environments.

  • You're writing your own software.

  • And you want to make use of recent advances

  • in machine learning.

  • You've read about this model.

  • You saw this YouTube video about this model.

  • Can you use it in your thing?

  • Well, before Runway one of the things you might have done

  • is find your way to some GitHub repo that

  • had like this very long ReadMe about all

  • the different dependencies you need to install and configure.

  • And then you've got to download this and install this, and then

  • build this library.

  • And you can really get stuck there for a long time.

  • So Runway is an all in one piece of software

  • with an interface that basically will run machine learning

  • models for you, install and configure them

  • without you having to do any other work

  • but press a button called Install.

  • And it gives you an interface to play with those models,

  • experiment with those models, and then broadcast

  • the results of those models to some other piece of software.

  • And there's a variety of ways you

  • can do that broadcasting, through HTTP requests,

  • through OSC messages.

  • And all these things might not make sense

  • to you, which is totally fine.

  • I am going to poke through them and show you

  • how they work, with an eye towards at least showing you

  • how to pair Runway with processing,

  • and how to pair Runway with P5JS,

  • and I'll also show you where there's lots of other examples

  • and things you can do with other platforms, and stuff like that.

  • So the first step you should do is click here

  • under Download Runway Beta.

  • It will automatically trigger a download

  • for Mac OS, Windows, or Linux.

  • I've actually already downloaded and installed Runway.

  • So I'm going to kind of skip that step,

  • and just actually now run the software.

  • Ah.

  • And now it's saying, welcome to Runway.

  • Sign in to get started.

  • OK.

  • So if you already have an account,

  • you could just sign in with your account.

  • I do already have an account.

  • But I'm going to create a new one, just so we can

  • follow along with the process.

  • So I'm going to go here.

  • Create an account.

  • I'm going to enter my email address, which is-- shh.

  • Don't tell anyone-- daniel@thecodingtrain.com.

  • Then I'm going to make a username and password.

  • Now that I've put in my very strong password,

  • I'm going to click Next.

  • And I'm going to give my details, Daniel Schiffman,

  • The Coding Train.

  • Create account.

  • Ah.

  • And it's giving me a verification code

  • to daniel@thecodingtrain.com.

  • Account has now been created, and I can click Start.

  • So once you've downloaded, installed Runway, and signed up

  • for an account, logged into your account,

  • you will find this screen.

  • So if you've been using Runway for a while,

  • you might then end up here, clicking on open workspaces,

  • because workspaces are a way of collecting

  • a bunch of different models that you

  • want to use for a particular project into a workspace.

  • But we haven't done any of that.

  • So the first thing that I'm going to do

  • is just click on Browse Models.

  • So the first thing that I might suggest that you do

  • is just click on a model and see what

  • you can do to play with it in the Runway interface itself,

  • because one of the things that's really wonderful about Runway

  • is as a piece of software and an interface you can explore

  • and experiment with the model to understand how it works,

  • what it does well, what it doesn't do well,

  • what it does at all, before starting

  • to bring it into your own software or your own project.

  • So I'm going to pick this Spade Coco model, which I have never

  • looked at before.

  • This is very legitimate me.

  • I have no idea what's going to happen when I click on that.

  • And now, here I can find out some more information

  • about the model.

  • So I could find out what does the model do?

  • It generates realistic images from sketches and doodles.

  • I can find out more information about the model.

  • For example, this is the paper that describes this model,

  • "Semantic Image Synthesis with Spatially Adaptive

  • Normalizations Trained on COCO-Stuff Data Set."

  • Remember when someone asked, is this a tutorial for beginners.

  • Well, it is for beginners in that you're a beginner.

  • You can come here and play around with it.

  • But you can go very deep too if you want to find the paper,

  • read through the notes, and understand

  • more about this model, how it was built,

  • what data it was trained on, which is always

  • a very important question to ask whenever you're

  • using a machine learning model.

  • So we can see there are attributions here.

  • So this is the organization that trained the model.

  • These are the authors of the paper.

  • We can see the size of it, when it was created,

  • if it's CPU and GPU supported.

  • We could also go under Gallery.

  • And we can see just some images that have been created.

  • So we can get an idea.

  • This is a model that's themed around something

  • called image segmentation.

  • So I have an image over here.

  • What does it mean to do image segmentation?

  • Well, this image is segmented, divided into a bunch

  • of different segments.

  • Those segments are noted by color.

  • So there's a purple segment, a pink segment,

  • a light green segment.

  • And those colors are tied to labels in the model,

  • essentially, that know about a kind of thing

  • that it could draw in that area.

  • So you could do image segmentation in two ways.

  • I could take an existing image, like an image of me,

  • and try to say, oh, I'm going to segment it.

  • This is where my head is.

  • This is where my hand is.

  • This is where my hand is.

  • Or I could generate images by sort

  • of drawing on a blank image, saying put a hand over here.

  • Put a head over here.

  • So that's what image segmentation

  • is, at least in the way that I understand it.

  • What have I done so far?

  • I've downloaded Runway.

  • I've poked around the models.

  • And I've just clicked on one.

  • Now, I want to use that model.

  • I want to play with it.

  • I want to see it run.

  • So I'm going to go here to Add to Workspace.

  • It's right up here.

  • Add to Workspace.

  • Now, I don't have a workspace yet.

  • So I need to make one.

  • And I'm going to call this workspace,

  • I'm going to say Coding Train Live Stream.

  • So I'm going to do that.

  • I'm going to hit Create.

  • Now, I have a workspace.

  • You can see, this is my workspace.

  • I have only one model added to this workspace over here.

  • And it's kind of highlighting up for me right now what to do.

  • I need to choose an input source.

  • So every machine learning model is different.

  • Some of them expect text input.

  • Some of them expect image input.

  • Some of them might expect input that's

  • arbitrary scientific data from a spreadsheet.

  • Then the model is going to take that input in, run it

  • through the model, and produce an output.

  • And that output might be numbers.

  • Or it also might be an image.

  • Or it might be more text.

  • So now we're in sort of the space of a case by case basis.

  • But if I understand image segmentation correctly,

  • I'm pretty sure the input and the output

  • are both going to be an image.

  • Let's make a little diagram.

  • So we have this--

  • what was this model called again?

  • Spade Coco.

  • So we have this machine learning model.

  • Presumably there's some neural network architecture in here.

  • Maybe it has some convolutional layers.

  • This is something we would want to read that paper

  • to find out more.

  • Runway is going to allow us to just use it out of the box.

  • And I certainly would always recommend

  • reading more about this to learn more about how to use it.

  • So my assumption here is in my software that I want to build,

  • I want to maybe create a drawing piece of software

  • that allows a user to segment down an image.

  • So you can imagine maybe I'm going to kind of draw

  • something that's one color.

  • Look, I could use different colored markers.

  • I'm going to sort of fill this image in with a bunch

  • of different colors.

  • And then I am going to feed that into the model.

  • And out will come an image.

  • So we have input.

  • And we have output.

  • And again, this is going to be different for every model

  • that we might pick in Runway.

  • Although, there's a lot of conventions.

  • A lot of the models expect images

  • as input and output images.

  • Some of them expect text as input, and output an image,

  • or image as input and output text.

  • Et cetera.

  • And so on and so forth.

  • And so now what I want to do is choose the input source

  • in Runway for the model.

  • So something that's going to produce a segmented image.

  • So that could be coming from a file.

  • It could actually come from a network connection, which

  • I'll get into maybe in a future video,

  • or you can explore on your own.