My name is Jared.

I'm an engineer on the TensorFlow Lite team.

Today I will be giving a very high level overview

with a few deep dives into the TensorFlow Lite

stack, what it is, why we have it, what it can do for you.

Again, this is a very broad topic.

So there will be some follow up here.

And if you have any questions, feel free to interrupt me.

And you know, this is meant to be enlightening for you.

But it will be a bit of a whirlwind.

So let's get started.

First off, I do want to talk about some

of the origins of TensorFlow Light

and what motivated its creation, why we have it

in the first place and we can't just use TensorFlow on devices.

I'll briefly review how you actually use TensorFlow Lite.

That means how you use the converter.

How you use the runtime.

And then talk a little bit about performance considerations.

How you can get the best performance on device

when you're using TensorFlow Lite.

OK.

Why do you need TensorFlow Lite in your life?

Well, again, here's some kind of boilerplate motivation

for why we need on device ML.

But these are actually important use cases.

You don't always have a connection.

You can't just always be running inference in the cloud

and streaming that to your device.

A lot of devices, particularly in developing countries,

have restrictions on bandwidth.

They can't just be streaming live video

to get their selfie segmentation.

They want that done locally on their phone.

There's issues with latency if you need

real time object detection.

Streaming to the cloud, again, is problematic.

And then there's issues with power.

On a mobile device, often the radio

is using the most power on your device.

So if you can do things locally, particularly with a hardware

backend like a DSP or an MPU, you

will extend your battery life.

But along with mobile ML execution,

there are a number of challenges with memory constraints,

with the low powered CPUs that we have on mobile devices.

There's also a very kind of fragmented and heterogeneous

ecosystem of hardware backends.

This isn't like the cloud where often

you have a primary provider of your acceleration backend

with, say, NVIDIA GPUs or TPUs.

There's a large class of different kinds

of accelerators.

And there's a problem with how can we

actually leverage all of these.

So again, TensorFlow works great on large well-powered devices

in the cloud, locally on beefy workstation machines.

But TensorFlow Lite is not focused on these cases.

It's focused on the edge.

So stepping back a bit, we've had TensorFlow

for a number of years.

And why couldn't we just trim this down

and run it on a mobile device?

This is actually what we call the TensorFlow mobile project.

And we tried this.

And after a lot of effort, and a lot of hours,

and blood, sweat, and tears, we were

able to create kind of a reduced variant of TensorFlow

with a reduced operator set and a trimmed down runtime.

But we were hitting a lower bound

on where we could go in terms of the size of the binary.

And there was also issues in how we

could make that runtime a bit more extensible,

how we could map it onto all these different kinds

of accelerators that you get in a mobile environment.

And while there have been a lot of improvements

in the TensorFlow ecosystem with respect to modularity,

it wasn't quite where we needed it

to be to make that a reality.

AUDIENCE: How small a memory do you need to get to?

JARED DUKE: Memory?

AUDIENCE: Yeah.

Three [INAUDIBLE] seem too much.

JARED DUKE: So this is just the binary size.

AUDIENCE: Yeah.

Yeah.

[INAUDIBLE]

JARED DUKE: So in app size.

In terms of memory, it's highly model dependent.

So if you're using a very large model,

then you may be required to use lots of memory.

But there are different considerations

that we've taken into account with TensorFlow Lite

to reduce the memory consumption.

AUDIENCE: But your size, how small is it?

JARED DUKE: With TensorFlow Lite?

AUDIENCE: Yeah.

JARED DUKE: So the core interpreter runtime

is 100 kilobytes.

And then with our full set of operators,

it's less than a megabyte.

So TFMini was a project that shares

some of the same origins with TensorFlow Lite.

And this was, effectively, a tool chain

where you could take your frozen model.

You could convert it.

And it did some kind of high level operator fusings.

And then it would do code gen. And it would kind of

bake your model into your actual binary.

And then you could run this on your device and deploy it.

And it was well-tuned for mobile devices.

But again, there are problems with portability

when you're baking the model into an actual binary.

You can't always stream this from the cloud

and rely on this being a secure path.

And it's often discouraged.

And this is more of a first party solution

for a lot of vision-based use cases and not a general purpose

solution.

So enter TensorFlow Lite.

Lightweight machine learning library

from all embedded devices.

The goals behind this were making ML easier,

making it faster, and making the kind of binary size and memory

impact smaller.

And I'll dive into each of these a bit more in detail

in terms of what it looks like in the TensorFlow Lite stack.

But again, the chief considerations

were reducing the footprint in memory and binary size,

making conversion straightforward,

having a set of APIs that were focused primarily on inference.

So you've already crafted and authored your models.

How can you just run and deploy these on a mobile device?

And then taking advantage again of mobile-specific hardware

like these ARM CPUs, like these DSP and NPUs

that are in development.

So let's talk about the actual stack.

TensorFlow Lite has a converter where

you ingest the graph def, the saved model, the frozen graphs.

You convert it to a TensorFlow Lite specific model file

format.

And I'll dig into the specifics there.

There's an interpreter for actually executing inference.

There's a set of ops.

We call it the TensorFlow Lite dialect

of operators, which is slightly different than

the core TensorFlow operators.

And then there's a way to plug in these different hardware

accelerators.

Just walking through this briefly, again,

the converter spits out a TFLite model.

You feed it into your runtime.

It's got a set of optimized kernels

and then some hardware plugins.

So let's talk a little bit more about the converter itself

and things that are interesting there.

It does things like constant folding.

It does operator fusing where you're

baking the activations and the biased computation

into these high level operators like convolution, which

we found to provide a pretty substantial speed up

on mobile devices.

Quantization was one of the chief considerations

with developing this converter, supporting

both quantization-aware training and post-training quantization.

And it was based on flat buffers.

So flat buffers are an analog to protobufs, which are

used extensively in TensorFlow.

But they were developed with more real time considerations

in mind, specifically for video games.

And the idea is that you can take a flat buffer.

You can map it into memory and then read and interpret that

directly.

There's no unpacking step.

And this has a lot of nice advantages.

You can actually map this into a page and it's clean.

It's not a dirty page.

You're not dirtying up your heap.

And this is extremely important in mobile environments

where you are constrained on memory.

And often the app is going in and out of foreground.

And there's low memory pressure.

And there's also a smaller binary size

impact when you use flat buffers relative to protobufs.

So the interpreter, again, was built from the ground

up with mobile devices in mind.

It has fewer dependencies.

We try not to depend on really anything at base.

We have very few absolute dependencies.

I already talked about the binary size here.

It's quite a bit smaller than--

the minimum binary size we were able to get with TensorFlow

Mobile was about three megabytes for just the runtime.

And that's without any operators.

It was engineered to start up quickly.

That's kind of a combination of being able to map your models

directly into memory but then also having a static execution

plan where there's--

during conversion, we basically map out

directly what the sequence of nodes that would be executed.

And then for the memory planning,

basically there's a pass when you're running your model where

we prepare each operator.

And they kind of cue up a bunch of allocations.

And those are all baked into a single pass where we then

allocate a single block of memory and tensors

are just fused into that large contiguous block of memory.

We don't yet support control flow.

But I will be talking about that later in the talk.

It's something that we're thinking about and working on.

It's on the near horizon for actual shipping models.

So what about the operator set?

So we support float and quantized types

for most of our operators.

A lot of these are backed by hand-tuned, neon, and assembly

based kernels that are specifically

optimized for ARM devices.

Ruy is our newest GEMM backend for TensorFlow Lite.

And it was built from the ground up with mobile execution

in mind, a [INAUDIBLE] execution.

We support about 120 built-in operators right now.

You will probably realize that that's quite

a bit smaller than the set of TensorFlow ops, which

is probably into the thousands by now.

I'm not exactly sure.

So that can cause problems.

But I'll dig into some solutions we have on the table for that.

I already talked about some of the benefits

of these high level kernels having

fused activations and biases.

And then we have a way for you to kind of, at conversion time,

stub out custom operators that you would like.

Maybe we don't yet support them in TF Lite

or maybe it's a one off operator that's not yet

supported in TensorFlow.

And then you can plug-in your operator implementation

at runtime.

So the hardware acceleration interface,

we call them delegates.

This is basically an abstraction that

allows you to plug in and accelerate

subgraphs of the overall graph.

We have NNAPI, GPU, EdgeTPU, and DSP backends on Android.

And then on iOS, we have a metal delegate backend.

And I'll be digging into some of these and their details

here in a few slides.

OK.

So what can I do with it?

Well, I mean this is largely a lot of the same things

that you can do with TensorFlow.