Name: TFLite Delegatesでモデルを高速化 (TF Dev Summit '20) (Accelerate models with TFLite Delegates (TF Dev Summit '20))
Uploaded: 2021-01-14T10:37:28.000Z
Duration: 15 min 20 s
Description: VoiceTubeの動画で発音を聞きながら英語表現を覚えよう！学べる英語：

I'm Sachin from the TensorFlow Lite team,

like to go over some of the basics of what delegation is.

Typically, a user would start with a TensorFlow model

and use a converter to convert the model into the TFLite

This TFLite file would then be handed down

to our interpreter, which runs the model on device.

to our CPU Op Kernels that are highly optimized for the ARM

However, most devices these days, especially mobile phones,

have a lot of other chips, like mobile GPUs or DSPs.

between the TensorFlow Lite runtime and lower level

as an interface between TensorFlow Lite and Android's

Or the GPU delegate uses OpenCL and OpenGL

to run inference on mobile GPUs on Android devices.

A natural question here is why would you use delegates at all.

The most obvious benefit is foster inference.

The classic example here is the GPU delegate.

Because of the highly parallelized nature of the GPU,

math, such as convolutions or fully connected layers.

As a result, when we use our GPU delegate with TensorFlow Lite,

we observe up to seven speed ups with a lot of division models

that are currently used on mobile devices.

Another great benefit is lower power consumption.

A good example here is the DSP, or the digital signal

DSPs are meant for applications such as multimedia

and communication, which inherently require less power

used our delegates that leverage Qualcomm's Hexagon DSP to run

even some of the mobile optimized models,

such as the MobileNet or the MobileNet SSD.

Now, suppose you have your own secret accelerator,

and you want to use our delegate API to write your own delegate.

So the bulk of how the interpreter delegates nodes

is in this function that we like to call DelegatePrepare.

This function gets an object called the TFLite context,

which is essentially an interface into the TensorFlow

nothing but a list of nodes that are going

such as what op it executes or what are the types of input

This has the delegate make an informed decision

about which ops it can accept to be delegated.

Once this list of supported nodes is populated,

and the other is what we call the kernel administration.

But let's look at what the runtime does when

Here we have a simple example of a model of Add and Mul ops,

and let's say our delegate only supports the Add operation.

So once the delegate calls this method with the nodes,

the runtime partitions all the nodes into two subsets,

One is delegated, and the other is non-delegated.

Now, there are two reasons why this happens.

First is many of the delegates optimize inference

So this way, the delegate can maximize the fusings and fuse

Another great reason is that it's very expensive

to go back and forth between the CPU and the accelerator,

Once this partitioning is done, the TensorFlow Lite runtime

At this point, the delegate op behaves just

like any other TFLite node for the runtime.

up is what is defined by the kernel implementation,

or the TFLite illustration that we saw in the previous slide.

字幕リスト動画再生

TFLite Delegatesでモデルを高速化 (TF Dev Summit '20) (Accelerate models with TFLite Delegates (TF Dev Summit '20))

essentially

context

figure

improve