Placeholder Image

字幕表 動画を再生する

  • Hello and welcome to a coding challenge.

  • Oh, my ukulele is very out of tune.

  • What luck this happens to be the make a ukulele tuner

  • coding challenge.

  • What a coincidence.

  • Well, I'm here in my new studio located--

  • it's not my studio, but it is a studio.

  • It's in Brooklyn at New York University.

  • And I'm doing my first coding challenge from here.

  • I don't really know what makes sense.

  • If this is a moment to do some sort of particular coding

  • challenge, I don't know.

  • I'm just going to do we're going to tune

  • the ukulele because the tuner I have, the battery is broken.

  • My ears are not so good.

  • So we'll see if this works.

  • I'm going to use the ml5.js library.

  • This is a JavaScript library that I'm lucky enough

  • to participate in its development

  • and I try to make tutorials with, so this works well.

  • It has a pretrained model for pitch detection, which

  • I'll talk about in a moment.

  • And then I'm also going to use the p5 Web Editor.

  • It'll be a little tricky to do this in the p5 Web Editor

  • because it's going to involve having

  • to upload a bunch of stuff, but the good news

  • is I already have the ml5 library right here.

  • All you need to do if you're in the p5 Web

  • Editor at editor.p5js.org, if you go to the ml5 website

  • under Getting Started, click on this link, which will open

  • a sketch in the p5 Web Editor with the ml5 library already

  • imported.

  • OK, so we're there.

  • I can make sure this is working by saying something like,

  • console.log ml5.version.

  • I'm going to hit Run.

  • And I see the version down here, so I've got ml5 going.

  • Now, what do I need to do next?

  • So if I go to the ml5 website, I'm going to Reference.

  • And in Reference, I can see that the various functionality

  • of ml5 is divided into different categories based on media.

  • And what I definitely want to do is look at sound

  • here because I want to find a sound model.

  • And lo and behold, there is a model called pitchDetection.

  • Now, I should mention that there are

  • ways of analyzing a sound for pitch that you don't

  • need machine learning for.

  • You could do FFT analysis and look at the different various

  • amplitude of different frequencies and pick the one

  • that's--

  • there's a variety of ways.

  • And people much smarter who know much more about sound

  • could tell you how to do that.

  • And I'm sure you could find other tutorials, but I'm here.

  • I want to try to use the ml5 library.

  • But this really begs the question,

  • like, well, how is this working?

  • You said something about pretrained model.

  • So ml5 comes with a certain model known as CREPE.

  • I don't know if that's how you pronounce it.

  • I like to say crepe, which is a Convolutional Representation

  • for Pitch Estimation or CREPE.

  • I like fruit-- a little banana--

  • and maybe a little Nutella is kind of good.

  • I don't know.

  • It's too much for me, the Nutella.

  • This is about pitch detection though.

  • That's what this video is about.

  • And so you can read a lot more about this particular model

  • and what data was used to train it, which is always

  • a question you should ask when you're using someone's

  • pretrained model because there's a lot of things that

  • can go wrong, or right, or be problematic potentially

  • about a model based on the data it was trained.

  • And this paper describes that in more detail.

  • You could click on this link over here to see a demo of it

  • in the browser, but we're going to do this in ml5.

  • And a big thank you to Hannah Davis,

  • who actually did the porting of this model into ml5.

  • And I'll include some links to her work

  • in this video's description.

  • OK, so here I am on the ml5 documentation page.

  • And it looks a little bit like, what's going on here?

  • So I need to create a pitch object.

  • And there's some sort of, like, string ./model/.

  • What's that?

  • First thing I want to tackle here is what this ./model/ is.

  • So a lot of times when using ml5,

  • it's going to load the model files from a URL.

  • And you might actually put the URL into your code

  • or ml5 might just know the URL automatically-- it's

  • saved on a Google server or some other server,

  • it's saved on GitHub and pointed to by ml5.

  • In this case, this is a case where I actually

  • need to have the model files with my code.

  • This was probably going to change.

  • Just by making this video, I've realized we probably

  • should host a version of the CREPE model

  • that you could access through ml5 more easily.

  • But luckily, if I go to the ml5-data-and-models GitHub

  • repo, where I am right here-- github.com/ml5js.

  • I can navigate to models/pitch-detection/crepe.

  • And these are all the model files.

  • So this is very typical of any pre-trained machine

  • learning model.

  • There will be a JSON file.

  • This is essentially a file that describes the model.

  • And then there's all these other files,

  • which are the actual weights, the numbers,

  • the secret sauce of the model after it's been trained.

  • All of those little parameters of the neural network

  • are all stored in all of these files.

  • Now, I've actually downloaded all of these already.

  • And you can see them.

  • They're right here in a folder on the desktop here.

  • So what I want to do right now is add all of them

  • to the P5 web editor.

  • Let's see how that goes.

  • I've really only worked with uploading little media files

  • and sound files, but I think this is going to work.

  • So I'm going to go here.

  • I'm going to create a new-- it's hard for you to see this,

  • but I'm going to create a new folder.

  • And I'm going to call this crepe.

  • And then I'm going to do Add File.

  • And I'm hoping that I could just select all of these

  • and drag them here.

  • So I forgot that the web editor currently only supports

  • certain file types, like JSON, or CSV, or JPEG image

  • files, sound files.

  • So these model files that include all the weights,

  • they can't be uploaded to the web editor.

  • And that's something that might change in the future.

  • But luckily, I can actually just point to the model files

  • that are on GitHub itself.

  • So this particular URL right here

  • where all these model files are stored,

  • there's actually a way to turn any file that's

  • sitting on GitHub into a URL that you

  • can load from a content delivery network.

  • And so a way of doing that, this is a nice blog

  • post that I found on gomakethings.com

  • that just shows this base URL.

  • So I can always access files through this URL,

  • cdn.jsdelivr.net/gh for GitHub, and then the path to the user

  • name, the repo, and the path of the files.

  • So I actually have done that right here.

  • I'm going to hit Refresh.

  • And you can see look, this is that model.json.

  • file.

  • And now, I can actually look and see,

  • oh, it's all of the configuration information

  • about this particular model.

  • And I can grab this, and I can put this into my code.

  • So let me close this.

  • I'm going to go to the top.

  • I'm going to say const modelurl equals,

  • and I'm going to paste that in there.

  • So now, I actually want just the path,

  • because I want the model to load all the files.

  • So I'm going to then remove model.json.

  • And this is the path to the crepe model.

  • And I can go ahead and just delete this

  • from here, this folder.

  • It's gone.

  • I guess I need to also delete the files one at a time.

  • Let me do that.

  • And now, I am ready to start putting in some code.

  • So I'm going to make a variable called pitch detector.

  • Let's just call it pitch.

  • I'm going to say pitch equals ML5 pitch detection.

  • Let me give myself some more space.

  • Pitch detection.

  • I'm going to go back to the reference page.

  • And these are the things that I need to load.

  • So let me copy-paste.

  • These are the parameters that I need to pass in.

  • I'm sitting here wondering why I have an error.

  • And of course, if I declare a variable as a const,

  • I can't just have it not equal anything and then assign it

  • later.

  • So this is going to have to be let.

  • And I'm going to make a lot of people angry right

  • now by making everything let just to simplify things.

  • And now, I have the--

  • I've now created a pitch detection object.

  • Now, what I want is to give it a bunch of arguments

  • to create itself, one of which is the model itself.

  • So this is no longer a local directory of files.

  • I'm going to say modelurl.

  • I need to get this audio context and mic stream.

  • Let me come back to that.

  • But I also need a model loaded function,

  • so that I know that the model has been loaded.

  • All right.

  • So audio context and mic stream, what are those things?

  • Well, let's hope the documentation tells us.

  • Audio context, the browser audio context to use.

  • Stream, media stream, the media stream to use.

  • I'm a little bit lost, to be honest.

  • Can I get some more information?

  • You know what I think we might do here is

  • let's just look at the example.

  • So the example here, if I look at the example code,

  • is showing, ah, perfect.

  • So an audio context I can just get by saying

  • get audio context.

  • Perfect.

  • So this happens to be something that's built into JavaScript.

  • It's part of the Web Audio API, I would assume.

  • And I'm sure somebody in the chat or someone

  • will leave a comment to explain what this is a little bit more.

  • But I'm just going to go ahead and put it in here.

  • I'm going to say audio context equals get audio context.

  • And then for the mic stream, this

  • is actually me connecting to the built-in microphone.

  • Or potentially, I could specify a different microphone.

  • And that I'm going to connect by using the P5 Sound Library.

  • So I can make a variable called mic,

  • and then I can say mic equals P5 Audio In, I

  • believe is the function.

  • So this is the function.

  • This is part of p5sound.js, which

  • incidentally is a library that I am accessing here

  • in indexed HTML.

  • And look at this.

  • I am on such an old version of P5.

  • Let's update this stuff.

  • And I think the current version is 9.0.

  • So while I'm here, I'm going to update that, go back to here.

  • And then I think I can get this mic stream from the P5 Mic

  • Object.

  • I'll just look it up here, mic.stream.

  • Perfect.

  • So this P5 audio-- oh, and I need to say mic.start,

  • start pitch.

  • I don't see any function.

  • Oh, interesting.

  • Look at this.

  • So I wasn't paying close attention.

  • I didn't really think what I'm doing, because I've

  • got to do two things here.

  • I need to load them--

  • I need to access the microphone, and I need to load the model.

  • When I load the model, I want to connect it to the microphone.

  • And maybe I need to think about the sequence going on here,