Thomas Drach KATIE: hello? There we go. That's me. Good

morning. Whoo! Is everybody ready? Yeah. We have a really, really cool talk to start the

day with. Thomas Drach is here. And when I, you know, he gave me some fun facts. But actually

we had a really fascinating conversation just now that I'm gonna share with you. So, back

in the '90s, the first two CDs that I ever bought. The first was DJ Jazzy Jeff and the

Fresh Prince, the rapper, parents don't understand, which was amazing. And I bought a Rick Astley CD

with a famous song on it that people use to Rickroll each other. And Thomas said his first

tape was Nsync. Clearly his musical taste is much better than me. Let's give it up for

Thomas Drach. [ Applause ]

THOMAS: Good morning. Thanks for being here early for this talk. I want to give a huge

thanks to JSConf for having me. They have been awesome. So grateful to be here. I also

want to thank any open source contributors. Because I feel like I'm using cheat code sometimes,

just like using someone else's code. If you're anything like me, I may have squirmed a little

bit at this AI acronym. And that's good. We're going through that today. But just your willingness

to be here, shows you're open to the ideas and pushing the boundaries. I think that's

commendable. Thanks for being here. Okay. My name's Thomas Drach like we talked about.

Thomasdrach on Twitter if you want to bug me there, please do. I consider myself a designer

and a bit of a hacker. Not in the sense of this awesome movie. But in the writing bad

code sense. I'm really good at that. I have a little design studio called Subtract where

I make what I hope are useful products. One of which we'll go through today. And I have

a product called Cleverstack too if you want to check that out. So, I want to start with

a man called Paul Thomas. He had a little garage in Phoenix called the Thomas Brothers

Garage. Today people might call him an entrepreneur or a founder. But he was actually an inventor.

He filed for over a hundred patents and I'm still trying to track most of them down. I

found about a dozen of them. Most reminded me of the Rube Goldberg machines. I don't

know if you have seen these before. They're simple machines that create this weird trap

like mechanism. All the patents I found from him were complex machines like that. Like

nothing you would actually use. It's kind of hard to see here. But the name of this

one is panel manufacturing method. And it's basically a patent for this giant machine

that creates these like brick and concrete slabs and then we're supposed to ship them

to job are sites for people to build houses with them. Kind of made me sad a little bit

because it just seemed like a crazy person was documenting and patenting all of these

things. But the machines that get built. This is the one that we were just talking about.

The manufacturing machine to make the giant slabs. And they actually had to design and

patent these, like, semitrucks to ship them to job sites. They had these weird little

trucks that would like transport like these palettes of bricks everywhere. And clearly

they didn't invent the screw or the lever or the racket pinion or these things, but

they combined something to make something new and useful. And it's especially interesting

to me because this man was my great grandfather and my namesake. Some of you might be familiar

with this Henry Ford quote. The funny thing is, Henry Ford never said this. I did research

and the first time it was attributed to him was in 1999 in the cruise industry news quarterly.

And other people started using it and now people say it on stages like this. Sometimes

it's paired with like the Steve Jobs quote and kind of create this is genius complex

of, they didn't know what they want. We have to show them or whatever. But I think there's

something that people miss about this quote in particular. I think it resonates for a

reason. But my interpretation of this quote is, big progress isn't necessarily just like

an iteration of the last thing, but it's like a mutation of something that happened before.

Maybe a little bit like this. We could accidently combine a few unrelated things to find something

new. This is Tim Berners Lee talking about inventing the Internet. He said I just had

to take the hypertext idea and connect it to the TC P&D NS ideas and can be ta da, I

had the World Wide Web. There's an old LinkedIn, his profile page, it just said like web developer.

But he goes on this interview, and I recommend listening or reading interviews from him.

He goes on just to attribute all these other inventions and says if these didn't happen,

the Internet, at least I wouldn't have created it at that time. I don't know what would have

happened. So, the definition of mutation in the dictionary is the changing of structure,

resulting in a variant form that may be transmitted to subsequent generations. Hendrix famously

took right handed guitars, flipped them upside down and then eventually changed music. And

he did so in part because it was before Les Paul invented the electronic guitar. They

didn't invent it to invent it, but because they wanted the acoustic guitar louder. And

Grace Hopper, is one of the inventers of what we now call programming. A big reason we're

here today. And she started with knobs and switches on the Mark I. All of these were

mutations. Like it was different enough. Hopper's was a mutation. And I think AI is a built

of a mutation, at least how we talk about it today. There's much more data, advances

in machine learning, compute power thanks to Moore's Law. And it kind of created the

opportunity for something like AI to work. This is what I get when I search "AI" on Google.

I don't know about you. But this isn't very helpful for me. So, I'll ask a little bit

different question today. I want to ask, what are intelligent machines? We might be able

to define this. Just intelligence + machines. So, let's define intelligence. This is a quote;

I'm just going to read it really quick. People generally distrust the concept of machines

that approach and thus why not pass our own human intelligence. I think a lot of people

feel like this today. And this quote was actually written in 1970 in the book called the architecture

machine actually by the person who founded the MIT Media Lab. And it goes on to say that,

machines must be aware of their context in order to be intelligent. So, you can't have

like a machine without using the context, interacting with the world. It's not intelligent

in that case. There's no lack of context in the new Tesla roadsters. So, for our purposes,

and I'm just gonna say intelligence means using context. So, now we can define machines.

This should be pretty easy. We go to the dictionary and find a mechanically, electronically, or

electrically operated device for a task. Sounds good to me. Okay. So, with intelligence and

machines defined, I would like to introduce you to the concept of somewhat intelligent

machines. And this is what we're gonna build today. And this is just something that uses

context and rapidly completes something that a human could not. And we're gonna do all

of it in JavaScript. So, this is the actual machine instrument, musician, AI, whatever

you want to call it. This is what we're gonna build today. I'm going to walk through how

to generate drumbeats using pre trained machine learning models, APIs, libraries, stuff like

that. We're going to piece it together. And I find it a little bit hard for me to follow

tiny code so it's gonna be a little pseudo codey. Like I said, the first thing we needed

were a couple libraries we're going to use Magenta. If you haven't heard of Magenta already,

please check it out. It's incredible. A couple of people have talked about it already here

at JSConf. And then we're going to use Tone, which is actually a dependency Magenta which

gives us an easier to code interface for musical stuff. All right. Let's play some drums. This

is what the data structure for the drums will look like. You can set up a step sequence,

but this is a step sequence in Magenta. There's a pitch, there's an attribute that tells it's

a sample based pitch, not a tonal keyboard like thing. And there's quantization info.

There's a method that does that for you so you don't have to worry about it. Okay. So,

all we need to play that note sequence is two lines of code. We're going to create a

new instance in the Magenta music player. And I'm going to call it player.star on that.

And we're gonna get something like this. [drumbeats]

This is just our basic pattern that we plugged in. Right? It's not that exciting. We kind

of want something a little bit more like this. Like feed it in and we'll get something better

in the material. All right. But in order to do that and do super quick ML crash course.

I am not the one to go in depth about this. But let's all get on the same page. Okay.

So, usually write functions something like this. We want something, we put something

in, we want something back. Machine learning, it's a little bit more abstract, right? We

don't know abstractly like what we  how we would get there. Like, is this image a dog?

Here's the image. I don't know. Some of you might see 

[ Laughter ] Memes like this. This is one of my favorites.

Chicken fingers and the goldendoodles, I think. I don't know how dogs and cats became like

the hello world of machine learning. But I'm not mad about it. So, here's what we call

training data. If it was training data, it was probably labeled. So, this is a dog. I

probably should have said like fried chicken. That's not an actual chicken. So, you feed

all that to the machine. The machine says all these are dogs. They have this weird odd

thing on their face. We call that a feature. That feature to us looks like a nose. The

machine goes, okay, there's a nose. It's probably a dog. So, we feed them the image and it's

gonna guess, dog. All of these are just like probabilities. For our purposes, we want to

give it some drums and we want some better drums in turn. So, that's where Magenta comes

in. Magenta has a couple different models available. All of these are super cool and

it seems like they're coming out with more every week, every month. So, there's a MusicRNN

model, a Music VAE and a Piano Genie. Right now the Piano Genie is a VAI as well. Just

quick, RNN stands for recounter neural network, a bunch of nodes. It's like one of those.

But it loops through itself. And a VAE is a variational auto encoder. If you're familiar

with encoding and decoding, it works similarly to that. For our purposes, we're going to

use the MusicRNN models just in the context of Magenta. They have a little bit better

support for like individual instruments like drums. And this is kind of what that might

look like. So, if you have nodes on the network, you have it looping through itself and you

have an in and an out. For us, we're going to put in our initial drum beat and we're

going to expect a generated drum beat in return. Okay. So, we picked our MusicRNN models. This

is what the actual checkpoint is. So, this is like a pre trained model. Trained with

millions of drumbeats and it has a sense of what drumbeats are. There's a kick on one,

there's a snare on two or something like that. So, these are the three lines of code that

we need to generate a new drumbeat. So, you just create a new instance of our MusicRNN

model, the checkpoint that we had. We initialize the model, it loads itself up. And then we

call this method continue sequence. We feed it in our note sequence, feed the number of

steps which is kind of arbitrary. Could be 16, 32 or whatever. And then we feed it a

number from zero to two. We'll go over the temperature a little bit letter. So, after

we do that, we just get a sample in return and we play the same way we played the other

one. So, this is what that looked like. This is gonna be generated beat with a temperature

of 1.5. [[drumbeat]

And if you generate it again, it's going to come up with now beats that we've never heard.

Cool. So, yeah. [ Applause ] all right. That was cool. But

it was a little bit of a blackbox. So, I want to go through what happens when we call them

with a continue sequence. We call it here in the three lines are of code. All we're

gonna do here is what's happening behind the scenes is we're gonna convert the note sequence

which is that drum thing. We're going to convert it to a Tensor. And then we're going to encode

the tensor to match the model, the checkpoint that you have. If you're wondering what a

tensor is, you probably already know. If you remember municipal math, is scalars and vectors.

A tensor has three dimensions. That's why you hair the word shape when talking about

machine learning. And especially TensorFlow. These are all  but the last is tensor. And

then there's an internal method called sampleRNN. The inputs go into the TensorFlow library

and generates the next notes. If you want to get into the nitty gritty, TensorFlow.JS

is a great place to actually get your hands dirty there. It helps me to visualize like

this. Once more, the continueSequence. And the note here, the noteSequence, convert it

to a model, call the sampleRNN and get the new drums. I told you we were gonna talk about

temperature. It's interesting to me because it's one of the few inputs we have available.

Restructure train it with different drumbeats, which is kind of cool. But temperature is

like the level of entropy in the system. So, the lower the temperature, the more predictable

result we're gonna get. The higher, the less predictable it will be. So, just as an example,

and drop it down here to 0.2. Sounds like really similar to the original

drumbeat. And if we keep generating it, it's pretty much like the same, right? So, now

we're gonna try cranking it up to 1.5. [Drumbeats  ]

So, a little bit more exciting, for sure. This is the temperature I like. More fun.

And after we do that, we just have like a little demo button here. It will generate

a new file. And then sometimes what I will do is I will drop it in this, garage band

and use it as a musician for my band. If you're wondering why there's no audio right now,

it's because we're not judging my music skills today. We're talking about JavaScript. Okay.

So, that was cool. It was like almost somewhat intelligent. I wanted to take it one step