I'm working on Grappler and performance.

But today, I'm going to talk mostly

about how we do graph rewriting for functions.

So functions is a very important part of TF2.0.

It's very important for the end user, how user thinks

about the flow computation.

So it's not graph anymore.

It's a compositional functions.

But that creates a bunch of problems for us at runtime.

And so basically, we have to rewrite all the graph

and all of the functions to make it executable

and to avoid some specific problems

later comes with the functions.

So this is, on the left, this button code.

So this kind of V1 Python that people usually write.

So you define your variables, some constants.

You do assignment, and then you want

to return, update variables.

So if you don't add explicit control dependency,

you're going to read the initial values.

So you have to think what are your dependencies

and have to make everything very explicit.

And so on the right, this is some textural GraphDef

representation.

So there's not enough space to go to the full GraphDef

in the [INAUDIBLE].

So this is roughly how it will look

like if you go to the back string into TensorFlow.

If you get to GraphDef to the back string,

you can get some short representation of the graph,

something like on the right.

So variables are actually VarHandleOp.

This is a single kernel that returns DT results data type.

Constants, they are just simple Const nodes with the values.

And then when you [INAUDIBLE] add,

it is an AssignAdd and TensorFlow kernel

that takes results handle from VarHandleOp, and the constant,

and assigns and add a value.

And when you do it, it is another TensorFlow kernel

read variable op.

So you get TensorFlow [INAUDIBLE]

resource handle a and c.

And because you have explicit control dependencies,

this series depends on update a and update c, both of them.

And you fetch with a and c.

But explicitly, dividing control dependencies is very annoying.

So in TensorFlow 2.0, the same graph

will look something like this.

So you have your variables defined

outside of a function scope.

And you have a TensorFlow notation and_and_get.

So I want to add two different constants, two variables, a

and c, and get back our latest results.

So the pattern looks something like on the left.

So you don't have to explicitly add control dependencies, which

make lives easier.

And you don't have to think about program order.

So tf function traces the function.

And it has all the necessary control dependencies.

So when you execute a graph, you will

get the program order you would expect from [INAUDIBLE]..

So the FunctionDef, so the graph representation

looks something like on the right.

So we have a GraphDef.

And the GraphDef is basically a sequence of null Defs.

And we also have a FunctionDef.

It's a way to group together some subgraph with input

arguments and output arguments.

So this function on the left from the pattern

will be transformed to FunctionDef.

That will be represented, something like this.

So we have two input parameters over source type a and b.

And the return type of the function is pair of ints.

So the update a--

I'm sorry-- add.

We are assigning the new value to [INAUDIBLE] source.

And then we do [INAUDIBLE].

And we have the control dependencies

from the previous operation, touching the same resource.

And this is added automatically by TensorFlow 2.0

automatic control dependencies.

So we don't have to do anything.

And then we have written letters.

So we have functions at graph level,

at FunctionDef and GraphDef.

In TensorFlow, they have two types of returns.

You might return the data type.

So you return two Tensors, read a and read c.

But also, it has a special kind of return notation, control

return.

So it doesn't return any data.

But you might group some of the ops inside your FunctionDef.

And you can specify that these ops have a name.

And they must always run.

And when you have a control dependency on the function

call, run time verifies that every op

that is part of control [INAUDIBLE] will run.

And the main graph will look something like this.

So we have a partition call.

It is Def 2.0 function call mechanism

that does partitioning and multi-device function,

invocation, and graph optimization and all the other

[INAUDIBLE] that are required for the runtime.

And [INAUDIBLE] operations is just

the identity that is the first and the second output

of FunctionDef.

so?

Any questions?

Is it clear, like the notation on the right side?

SPEAKER 1: So in this example, in particular,

the read operations already depend on the updates.

So in this case, the control dependencies, I guess,

are not that required.

But in other case, if we don't return the read values then

those--

EUGENE ZHULENEV: No.

The read depends only on the input, a and c.

SPEAKER 1: Yeah but it has a control

dependency on the update.

EUGENE ZHULENEV: Yeah, because this control dependency

added automatically by Def 2.0 at Def function,

when you trace the function.

And you have multiple Def ops touching the same resource.

Automatic [INAUDIBLE] will add control dependencies.

SPEAKER 1: I'm saying in terms of the return values,

like in this case, if you have the read a and read

c, they automatically capture the--

so if you tried to fetch read a and read c,

you make sure that updates are run.

So in this case, the control dependencies

are not that useful, I guess, or are they--

SPEAKER 2: [INAUDIBLE]

SPEAKER 1: Yeah, yeah, the control returns.

EUGENE ZHULENEV: Oh, yeah.

Well, yes.

SPEAKER 1: Just pointing out that it

would be more useful if you don't return the right value.

Then in this case, the control returns would be very useful.

EUGENE ZHULENEV: Yeah.

Yeah, so if you have some way of update--

for example, if you have a function that

doesn't have returns, that you are a function only

for the side effects, update some counters,

much of counterstatistics.

In this case, you don't have irregular data outputs.

So this is why you have control return.

But right now, tf 2.0 will add these control returns

to both places.

This is needed because to separate the data path

and control path, because when you inline that function--

and it turns out that no one is using output 1.

So you might end up that you will not execute your state

updates inside the function.

But it is guaranteed that everything on the control path,

on the control region, will be executed under some conditions

that I'll mention later.

SPEAKER 3: Question.

What's that small [INAUDIBLE] in front of update [INAUDIBLE] a?

If you see the read a equals to ReadVariableOp in a, and then

what's that small sign in front?

EUGENE ZHULENEV: Control dependency, you mean?

This one?

SPEAKER 3: Yeah, this one.

Why it has this special--

EUGENE ZHULENEV: Yeah, so the data dependency goes by name.

So a is regular data dependency, and this one

is a control dependency.

So we don't read the actual output.

It just means that update a must execute

before we can execute read a.

SPEAKER 3: Oh, so it's just a special sign [INAUDIBLE]..

EUGENE ZHULENEV: Yeah.

Yeah.

SPEAKER 4: That's the internal representation

of control dependencies in the GraphDef total.

SPEAKER 3: I see.

EUGENE ZHULENEV: So yeah.

If your op has two inputs, you will have ab as inputs.

And then you can add as many as you want control dependencies.

At graph execution time, run we'll make sure

that everything in control dependencies

will be executed before your kernel with a special notation.

Here's another example.

So we have variables a and b, are vectors of length 10.

And variable c, it's just a counter, all types of integers.

And we want to get a strided slice from variable a and b.

And then we also want to increment the counter.

Every time we take a slice, we want

to know how many slices did you take for whatever reason.

So this graph doesn't make much sense.

But anyway, and we're only interested in slice 1.

And the graph will look something like this.

We have three variables, work handle ops, constant assigned

ad for the counter, and then we read a variable, this one.

So we read variables from--

this should be b.

So we read variables from their resource, from the handle,

and then we take a strided slice.

And then we fetch slice 1.

And because we fetch slice 1, we can remove slice 0.

It is not needed for slice 1.

We can also remove from the graph a, and that's all.

But you see that the nice property we get, that slice 0

is invalid.

If you tried to take a slice from 20 to 30,

in a variable of size 10, you'll get fail at run time.

But because you don't really need slice 0,

everything is just fine.

As long as you don't fetch slice 0, this graph is totally valid.

But if you do the same graph, but you put your slices

inside a function, so the same kind of thing,

you have three variables, a, b, c. c is a counter.

You do AssignAdd inside a function.

So nice, you don't have to add explicit control dependency.

You'll get your count updated automatically.

Then you take slice 0 and slice 1 and [INAUDIBLE]..

And you invoke a function.

You're not interested in slice 0.

You'll want only to get the second value.

So just keep it.

And you're trying to print it.

So at a GraphDef level and FunctionDef,

it will look something like this.

It will have a good slice function

that takes three sources, a, b, and c.

It will have an increment for the counter.

It will read the variants of a, b, take the slices.

And return slice 0 and slice 1.

And it will have a control [INAUDIBLE] increment

automatically added by automatic [INAUDIBLE] tracking.

So slices don't have any control dependencies

because slices, they do not depend on variable c.

They depend on different variables.

And then you have a function call,

a partition call to the get slice a and b.

And you don't need the first slice.

You take the second one by a function called output 1.

And you're trying to fetch it, or print it, or whatever.

And the problem right now, that this code,

we will fail because the function semantics [INAUDIBLE]

flow currently is strict with regards

to its inputs and outputs.

So before function return, function can return,

it must finish the execution of all the output parameters.

And before function can start executing,

it must finish executing of all the inputs.

Yeah, so there's another property.

So all inputs must be evaluated and must be not dead.

Your function can start running.

And note that it means that if you have inputs from switches,