CHRIS LATTNER: Hi, everyone.

I'm Chris.

And this is Brennan.

And we're super excited to tell you about a new approach

to machine learning.

So here in the TensorFlow team, it

is our jobs to push the state of the art

in machine learning forward.

And we've learned a lot over the last few years

with deep learning.

And we've incorporated most of that all into TensorFlow 2.

And we're really excited about it.

But, here, we're looking a little bit further

beyond TensorFlow 2.

And what do I mean by further?

Well, eager mode makes it really easy to train a dynamic model.

But deploying it still requires you take that and then write

a bunch of C++ code to help drive it.

And that could be better.

Similarly, some researchers are interested in taking machine

learning models and integrating them into larger applications.

That also often requires writing C++ code.

We always want more flexible and expressive autodifferentiation

mechanisms.

And one of things we're excited about

is being able to define reusable types that

then can be put into new places and used

with automatic differentiation.

And we always love improving your developer workflow.

We want to make you more productive

by taking errors in your code and bringing them

to your source and also by just improving your iteration time.

Now, what we're really trying to do here

is lift TensorFlow to entirely new heights.

And to do that, we need to be able to innovate

at all levels of the stack.

This includes the compiler and the language.

And that's what Swift for TensorFlow is all about.

We think that applying new solutions to old problems

can help push machine learning even further than before.

Well, let's jump into some code.

So first, what is Swift?

Swift is a modern and cross-platform programming

language that's designed to be easy to learn and use.

Swift uses types.

And types are great, because they can

help you catch errors earlier.

And also, they encourage good API design.

Now, Swift uses type inference, so it's really easy to use

and very elegant.

But it's also open source and has an open language evolution

process, which allows us to change the language

and make it better for machine learning which is really great.

Let's jump into a more relevant example.

This is how you define a simple model in Swift for TensorFlow.

As you can see, we're laying out our layers here.

And then we can find a forward function, which composes them

together in a linear sequence.

You've probably noticed that this looks a lot like Keras.

That's no accident, of course.

We want you to be able to take what you know about Keras

and bring it forward into this world as well.

Now, once we have a simple model, let's train it.

How do we do that?

All we have to is instantiate our model,

pick an optimizer and some random input data,

and then pick a training loop.

And, here, we'll write it by hand.

One of the reasons we like writing by hand

is that it gives you the maximum flexibility

to play with different kinds of constructs.

And you can do whatever you want, which is really great.

But some of the major advantages of Swift for TensorFlow

are the workflow.

And so instead of telling you about it, what do you think,

Brennan, should be show them?

BRENNAN SAETA: Let's do it.

All right, the team has thought long and hard

about what's the easiest way for people to get started

using Swift for TensorFlow.

And what could be easier than just opening up a browser tab?

This is Google Colab, hosted Jupyter notebooks.

And it comes with Swift for TensorFlow built right in.

Let's see it in action.

Here is the layer model, the model

that Chris just showed you a couple of slides ago.

And we're going to run it using some random training

data right here in the browser.

So we're going to instantiate the model.

We're going to use the stochastic gradient descent SGD

optimizer.

And here we go.

We have now just trained a model using

Swift for TensorFlow in our browser on some training data

right here.

Now, we can see the training loss is decreasing over time.

So that's great.

But if you're ever like me and whenever I try and use

machine learning in any application,

I start with a simple model.

And I've got to iterate.

I've got to tweak the model to make it fit better

to the task at hand.

So since we're trying to show you the workflow,

let's actually edit this model.

Let's make it more accurate.

So here we are.

Now, let's think a little for a moment.

What changes do we want to make to our model?

Well, this is deep learning after all.

So the answer is always to go deeper, right?

But you may have been following the recent literature in state

of the art in that not just sequential layers,

but skip connections or residual connections

are a really good idea to make sure your model continues

to train effectively.

So let's go through and actually add an extra layer

to our model.

Let's add some skip connections.

And we're going to do it all right now in under 90 seconds.

Are you ready?

All right, here we go.

So the first thing that we want to do

is we need to define our additional layer.

So we're going to fill in this dense layer.

Whoops.

Flow.

And one thing you can see is that we're

using Tab autocomplete to help fill

in code as we're trying to develop and modify our model.

Now, we're going to fix up the shapes right here really

quick, so that the residual connections will all work.

If I can type properly, that would go better.

All right, great.

We have now defined our model with the additional layers.

All we need to do is modify the forward pass,

so that we add those skipped connections.

So here we go.

The first thing we need to do is we

need to store in a temporary variable

the output of the flattened layer.

Then we're going to feed the output of the flattened layer

to our first dense layer.

So dense.applied to tmp in context.

Now, for the coup de grace, here is our residual connection.

So dense2.applied to tmp + tmp2 in context.

Run that.

And, yes, that works.

We have now just defined a new model

that has residual connections and is

one additional layer deeper.

Let's see how it does.

So we're going to reinstantiate our model

and rerun the training loop.

And if you recall from the loss that we saw before,

this one is now substantially lower.

This is great.

This is an example of what it's like to use Swift

for TensorFlow to develop and iterate as you apply models

to applications and challenges.

But Swift for TensorFlow-- thank

[APPLAUSE]

But Swift for TensorFlow was designed for researchers.

And researchers often need to do more than just change models

and change the way the architecture fits together.

Researchers often need to define entirely

new abstractions or layers.

And so let's actually see that live right now.

Let's define a new custom layer.

So let's say we had the brilliant idea

that we wanted to modify the standard dense layer that

takes a weights and biases and we

wanted to add an additional bias set of parameters, OK?

So we're going to define this double bias dense layer right

here.

So I'm going to type this really quickly.

Stand by 15 seconds.

Here we go.

[LAUGHTER]

Woo, all right, that was great.

So let's actually walk through the codes

that you can see what's going on.

So the first thing that we have is we define our parameters.

So these are a W, like our weights for our neurons,

and B1, bias one, and B2, our second bias.

We defined an initializer that takes

an input size and an output size just like dense does.

We use that to initialize our parameters.

The forward pass is very simple to write.

So here's just applied to.

And we just take the matrix multiplication of input

by our weights, and we add in our bias terms.

That's it.

We've now just defined a custom layer right in Colab

in just a few lines of code.

All right, let's see how it goes.

Here's model two.

And so we're going to use our double bias dense layer.

And we're going to instantiate and.

We're going to train it using, again, our custom

handwritten training loop.

Here's an example of another way that we

think Swift for TensorFlow makes your life easier.

Because Swift for TensorFlow can statically analyze your code,

it can be really helpful to you.

I don't know about you, but I regularly put typos in my code.

I don't if you saw me typing earlier.

And Swift for TensorFlow here is helping you out, right?

It's saying, look, you mistyped softmaxCrossEntropy.

This should be labels, OK?

All right, so we run it.

We train it.

And our loss isn't as good.

This was not the right idea.

But this is an example of how easy

it is for researchers to experiment with new ideas

really easily in Swift for TensorFlow.

But let's go deeper.

Swift for TensorFlow is, again, designed for researchers.

And researchers need to be able to customize everything, right?

That's the whole point of research.

And so let's show an example of how

to customize something other than just a model or a layer.

So you may have heard that large GPU clusters or TPU super pods

are, like, delivering massive breakthroughs in research

and advancing the state of the art in certain applications

and domains.

And you may have also heard that, as you scale up

to effectively utilize these massive hardware pools,

you need to increase your batch size.

And so let's say you're a researcher,

and you want to try and figure out

what are the best ways to train deep neural networks at larger

batch sizes.

Well, if you're a researcher, you probably

can't buy a whole GPU cluster or rent a whole TPU super pod all

the time for your experiments.

But you often have a GPU under your desk.

So let's see how we can simulate running on a super large data

parallel GPU or TPU cluster on a single machine.

We're going to do it all in a few lines of code right here.

So here's our custom training loop.

Well, here's the standard part, right?

This is 1 to 10 training epics.

And what we're going to do is, instead of just applying

our model forward once, we have an additional inner loop,