and I'll give you an intro to what that is.

My name's Andrew Selle.

I'm a software engineer down in Google in Mountain View land.

All right, so introduction.

We're going to be talking about machine learning on-device.

So how many of you are interested on running

your machine learning models on-device?

How many already do that?

OK, so a lot.

So that's great.

So you already know that machine learning on-device

is important.

So I'm going to give you a scenario that's

perhaps a little bit out of the ordinary.

Suppose I'm going camping.

I don't have any power.

I don't have any network connectivity.

I'm out in the mountains.

And I want to detect bears.

So I want to hang my cell phone on the outside of my tent

so that if I'm in the middle of the night

and the bear is coming for me, I'll know.

I don't know what to do if that happens,

but that's a great example of what you do with on-device ML.

So you basically want to have low latency.

You don't want to wait for the bear to already be upon you.

You want some early warning.

You want to make sure it works without a data connection.

The data can stay on the device and you have access

to the sensors.

So there's a lot more kind of practical and important use

cases than that.

But that kind of sets the stage for it.

And so people are, of course, doing a lot more ML on-device.

So that's why we started TensorFlow Lite.

So in short, TensorFlow Lite is our solution

for running on-device machine learning

with low latency and a small binary size

but on many platforms.

So here's an example of what TensorFlow Lite can do.

So let's play our video.

So, suppose we have these objects

and we want to recognize them.

Well, I know what they are, but I want my phone

to be able to do that.

So we use an image classification model.

So here we have a marker.

In the bottom you see kind of the confidence.

So this is an image recognition.

It's happening in real time.

It can do kind of ordinary office objects,

but also important objects like TensorFlow logos.

And it also runs on multiple platforms.

So here we have it running on an Android phone and an IOS phone.

But we're not just limited to phones.

We can also do things like this Android Things Toolkit.

And there is going to be a couple more later, which

I'll show you some more.

So now that we've seen it in action,

let's talk a little bit more about what TensorFlow Lite is.

Do you want to see the video again?

I don't know.

OK, there we go.

The unfortunate thing about on-device

though is it's much harder.

It has tight memory constraints, you need to be low energy,

and you don't have as much computation as you do available

on the cloud.

So TensorFlow Lite has a few properties

that are going to sort of deal with these problems.

So the first one is it needs to be portable.

You know, normal PCs we run on, but we also

run on mobile phones, we run on Raspberry Pi, or other Linux

SOCs, IoT type devices.

And we also want to go to much smaller devices

like microcontrollers.

So, next slide.

The internet doesn't like me today.

Well, we'll skip that slide, whatever that was.

But in any case, basically, this portability

is achieved through the TensorFlow Lite file format.

So once you have a trained TensorFlow model, like you

author, you could author it with Swift,

you could author it with Python, whatever you do,

you produce a saved model in a graph form.

The serialized form is then converted to TensorFlow Lite

format, which then is your gateway to running on all

these different platforms.

And we have one more special converter

which allows you to go to Core ML

if you want to target IOS in this especially way.

The second property is that it's optimizable.

We have model compression, we have quantization,

we have CPU kernel fusion.

These are all optimization techniques

to ensure that we have the best performance, as well

as a small size.

And this is achieved through the architecture.

So we have a converter, which we already talked about,

and then we have an interpreter core.

The interpreter core delegates to kernels

that know how to do things, just like in TensorFlow.

But unlike in TensorFlow these are

optimized for mobile and small devices with NEON on ARM.

An additional thing that we have that TensorFlow

doesn't have is the notion of delegation.

So we can delegate to GPUs or Edge TPUs or accelerators.

And this basically gives TensorFlow Lite

the chance to give part of the graph

to a hardware accelerator that can do processing

in a special way.

So one of those we've talked about before is in an API.

We're excited to announce that in 2018 Q4

we're looking to release our OpenGL-based GPU

delegate, which will give us better performance on GPUs,

and will also accelerate things like MobileNet

and another vision-based models.

So that's really exciting.

In addition, at Cloud Next there was an announcement

about Edge TPUs.

And Edge TPUs are also special as well

because they give us the ability to do

high performance per watt, and also

fit into a small footprint.

So for example, the device is there on that penny there,

but it's on a development board.

So you can put it in many different form factors as well.

Then the third property is that it's parametrically sized.

So we know the TensorFlow Lite needs

to fit on small devices, especially very small devices

like MCUs, And there, you might need to only include

the ops that you need.

So our base interpreter is about 80 kilobytes,

and with all built-in ops it's 750 kilobytes.

We're moving to a world where you can parameterize

what you put into the TensorFlow Lite interpreter,

so you can trade off the ability to handle

new models that use new ops, and the ability to only ship what

you need in your application.

So, we introduced TensorFlow Lite last year

and we've been asking users what they think of TensorFlow Lite.

And they really love the cross platform deployment

that they can deploy to IOS, to Android

with the same kind of format.

They like that they can decouple the distribution of the binary

from the distribution of the model.

And they like the inference speed increases.

And they're really excited about the hardware acceleration

roadmap.

But the biggest feedback that we got

is that we should focus on ease of use, we should add more ops,

and we should work on model optimization,

and we should provide more documentation.

And so we've listened.

So what I want to do in this talk

is focus on the user experience.

But before we do that, let's look

at some of the users that are already using it so far.

We have a lot of first party users

and a lot of third party users that are excited about it.

And I hope after this talk you'll be interested as well.

So, the user's experience.

I'm a new user and I want to use TensorFlow Lite.

How do I do it?

Well, I think of it as kind of like learning to swim.

You can think of two things you might do.

You might wade, where you don't really have to swim.

But it's really easy to get started

and you get to cool off.

The second thing is that you can swim

where you can do custom models.

So we're going to talk about both of those.

But before we get into that, there's

an easier thing and then a harder thing.

So the easier thing is to dip your toes, which are demos.

And the harder thing is to just dive full in

and have full kind of mastery of the whole water,

and that would be optimizing models.

And we'll talk about those as well.

So as far as using demo apps, you can go to our website,

you can download the demos and compile it and run it.

It'll give you a flavor of what can be done.

I showed you one of those demo apps.

You can try it for yourself.

The next step is to use a pre trained model.

So the demo app uses a pre trained model.

So you can use that model in your application.

So if you have something that could benefit from say ImageNet

style classification, you can just take that model

and include it.

Another thing that's really useful is retraining.

So let me show you a retraining workflow, which

is you take a pre trained model and you kind of customize it.

Great, so we're running this video.

We're showing you the scissors and the Post-It Notes

as before, and here's an application

that I built on PC that allows me to do retraining.

But we're running inference with TensorFlow Lite.

So it knows the scissors, it knows the Post-It Notes,

but what if we got to a really important object, one

that we haven't seen before, perhaps

like something everybody has, the a steel TensorFlow logo.

How is it going to do on that?

Well, not well is the unfortunate thing.

But the good thing about machine learning is we can fix it.

We just need some examples.

So here, this app allows us to collect examples.

And you could imagine putting this into a mobile app

where you just move your phone around

and it captures a bunch of examples.

I'm going to do the same thing, except on Linux.

It's a little bit more boring, but it does a job.

So we want to get a lot of examples

to get a lot of generalization.

So once we're happy with that, we'll hit the Train button

and that's going to do some machine learning.

And once it's converged, we're going to have a new model.

It's going to convert it to TensorFlow Lite, which

is going to be really great.

We can test it out.

See if it's now detecting this.

And indeed it is.

That's good.

The other cool thing about that is now

that we have this TF Lite flat buffer model,

we can now run it on our device and it works as well.

All right, great.

So, now that we've done pre train models and we've done--

let's get into full on swimming.

There's basically four steps that we need to work on.

The first one is building and training

the model, which we've already talked about.

You could do that with, again, Swift would be a great way

to do that.

The second one is converting the model.

Third one is validating the model.

And the fourth is deploying model.

Let's dive into them.

Well, we're not going to dive yet.

We'll swim into them.