Placeholder Image

字幕表 動画を再生する

  • EUGENE ZHULENEV: My name is Eugene.

  • I'm working on Grappler and performance.

  • But today, I'm going to talk mostly

  • about how we do graph rewriting for functions.

  • So functions is a very important part of TF2.0.

  • It's very important for the end user, how user thinks

  • about the flow computation.

  • So it's not graph anymore.

  • It's a compositional functions.

  • But that creates a bunch of problems for us at runtime.

  • And so basically, we have to rewrite all the graph

  • and all of the functions to make it executable

  • and to avoid some specific problems

  • later comes with the functions.

  • So this is, on the left, this button code.

  • So this kind of V1 Python that people usually write.

  • So you define your variables, some constants.

  • You do assignment, and then you want

  • to return, update variables.

  • So if you don't add explicit control dependency,

  • you're going to read the initial values.

  • So you have to think what are your dependencies

  • and have to make everything very explicit.

  • And so on the right, this is some textural GraphDef

  • representation.

  • So there's not enough space to go to the full GraphDef

  • in the [INAUDIBLE].

  • So this is roughly how it will look

  • like if you go to the back string into TensorFlow.

  • If you get to GraphDef to the back string,

  • you can get some short representation of the graph,

  • something like on the right.

  • So variables are actually VarHandleOp.

  • This is a single kernel that returns DT results data type.

  • Constants, they are just simple Const nodes with the values.

  • And then when you [INAUDIBLE] add,

  • it is an AssignAdd and TensorFlow kernel

  • that takes results handle from VarHandleOp, and the constant,

  • and assigns and add a value.

  • And when you do it, it is another TensorFlow kernel

  • read variable op.

  • So you get TensorFlow [INAUDIBLE]

  • resource handle a and c.

  • And because you have explicit control dependencies,

  • this series depends on update a and update c, both of them.

  • And you fetch with a and c.

  • But explicitly, dividing control dependencies is very annoying.

  • So in TensorFlow 2.0, the same graph

  • will look something like this.

  • So you have your variables defined

  • outside of a function scope.

  • And you have a TensorFlow notation and_and_get.

  • So I want to add two different constants, two variables, a

  • and c, and get back our latest results.

  • So the pattern looks something like on the left.

  • So you don't have to explicitly add control dependencies, which

  • make lives easier.

  • And you don't have to think about program order.

  • So tf function traces the function.

  • And it has all the necessary control dependencies.

  • So when you execute a graph, you will

  • get the program order you would expect from [INAUDIBLE]..

  • So the FunctionDef, so the graph representation

  • looks something like on the right.

  • So we have a GraphDef.

  • And the GraphDef is basically a sequence of null Defs.

  • And we also have a FunctionDef.

  • It's a way to group together some subgraph with input

  • arguments and output arguments.

  • So this function on the left from the pattern

  • will be transformed to FunctionDef.

  • That will be represented, something like this.

  • So we have two input parameters over source type a and b.

  • And the return type of the function is pair of ints.

  • So the update a--

  • I'm sorry-- add.

  • We are assigning the new value to [INAUDIBLE] source.

  • And then we do [INAUDIBLE].

  • And we have the control dependencies

  • from the previous operation, touching the same resource.

  • And this is added automatically by TensorFlow 2.0

  • automatic control dependencies.

  • So we don't have to do anything.

  • And then we have written letters.

  • So we have functions at graph level,

  • at FunctionDef and GraphDef.

  • In TensorFlow, they have two types of returns.

  • You might return the data type.

  • So you return two Tensors, read a and read c.

  • But also, it has a special kind of return notation, control

  • return.

  • So it doesn't return any data.

  • But you might group some of the ops inside your FunctionDef.

  • And you can specify that these ops have a name.

  • And they must always run.

  • And when you have a control dependency on the function

  • call, run time verifies that every op

  • that is part of control [INAUDIBLE] will run.

  • And the main graph will look something like this.

  • So we have a partition call.

  • It is Def 2.0 function call mechanism

  • that does partitioning and multi-device function,

  • invocation, and graph optimization and all the other

  • [INAUDIBLE] that are required for the runtime.

  • And [INAUDIBLE] operations is just

  • the identity that is the first and the second output

  • of FunctionDef.

  • so?

  • Any questions?

  • Is it clear, like the notation on the right side?

  • SPEAKER 1: So in this example, in particular,

  • the read operations already depend on the updates.

  • So in this case, the control dependencies, I guess,

  • are not that required.

  • But in other case, if we don't return the read values then

  • those--

  • EUGENE ZHULENEV: No.

  • The read depends only on the input, a and c.

  • SPEAKER 1: Yeah but it has a control

  • dependency on the update.

  • EUGENE ZHULENEV: Yeah, because this control dependency

  • added automatically by Def 2.0 at Def function,

  • when you trace the function.

  • And you have multiple Def ops touching the same resource.

  • Automatic [INAUDIBLE] will add control dependencies.

  • SPEAKER 1: I'm saying in terms of the return values,

  • like in this case, if you have the read a and read

  • c, they automatically capture the--

  • so if you tried to fetch read a and read c,

  • you make sure that updates are run.

  • So in this case, the control dependencies

  • are not that useful, I guess, or are they--

  • SPEAKER 2: [INAUDIBLE]

  • SPEAKER 1: Yeah, yeah, the control returns.

  • EUGENE ZHULENEV: Oh, yeah.

  • Well, yes.

  • SPEAKER 1: Just pointing out that it

  • would be more useful if you don't return the right value.

  • Then in this case, the control returns would be very useful.

  • EUGENE ZHULENEV: Yeah.

  • Yeah, so if you have some way of update--

  • for example, if you have a function that

  • doesn't have returns, that you are a function only

  • for the side effects, update some counters,

  • much of counterstatistics.

  • In this case, you don't have irregular data outputs.

  • So this is why you have control return.

  • But right now, tf 2.0 will add these control returns

  • to both places.

  • This is needed because to separate the data path

  • and control path, because when you inline that function--

  • and it turns out that no one is using output 1.

  • So you might end up that you will not execute your state

  • updates inside the function.

  • But it is guaranteed that everything on the control path,

  • on the control region, will be executed under some conditions

  • that I'll mention later.

  • SPEAKER 3: Question.

  • What's that small [INAUDIBLE] in front of update [INAUDIBLE] a?

  • If you see the read a equals to ReadVariableOp in a, and then

  • what's that small sign in front?

  • EUGENE ZHULENEV: Control dependency, you mean?

  • This one?

  • SPEAKER 3: Yeah, this one.

  • Why it has this special--

  • EUGENE ZHULENEV: Yeah, so the data dependency goes by name.

  • So a is regular data dependency, and this one

  • is a control dependency.

  • So we don't read the actual output.

  • It just means that update a must execute

  • before we can execute read a.

  • SPEAKER 3: Oh, so it's just a special sign [INAUDIBLE]..

  • EUGENE ZHULENEV: Yeah.

  • Yeah.

  • SPEAKER 4: That's the internal representation

  • of control dependencies in the GraphDef total.

  • SPEAKER 3: I see.

  • EUGENE ZHULENEV: So yeah.

  • If your op has two inputs, you will have ab as inputs.

  • And then you can add as many as you want control dependencies.

  • At graph execution time, run we'll make sure

  • that everything in control dependencies

  • will be executed before your kernel with a special notation.

  • Here's another example.

  • So we have variables a and b, are vectors of length 10.

  • And variable c, it's just a counter, all types of integers.

  • And we want to get a strided slice from variable a and b.

  • And then we also want to increment the counter.

  • Every time we take a slice, we want

  • to know how many slices did you take for whatever reason.

  • So this graph doesn't make much sense.

  • But anyway, and we're only interested in slice 1.

  • And the graph will look something like this.

  • We have three variables, work handle ops, constant assigned

  • ad for the counter, and then we read a variable, this one.

  • So we read variables from--

  • this should be b.

  • So we read variables from their resource, from the handle,

  • and then we take a strided slice.

  • And then we fetch slice 1.

  • And because we fetch slice 1, we can remove slice 0.

  • It is not needed for slice 1.

  • We can also remove from the graph a, and that's all.

  • But you see that the nice property we get, that slice 0

  • is invalid.

  • If you tried to take a slice from 20 to 30,

  • in a variable of size 10, you'll get fail at run time.

  • But because you don't really need slice 0,

  • everything is just fine.

  • As long as you don't fetch slice 0, this graph is totally valid.

  • But if you do the same graph, but you put your slices

  • inside a function, so the same kind of thing,

  • you have three variables, a, b, c. c is a counter.

  • You do AssignAdd inside a function.

  • So nice, you don't have to add explicit control dependency.

  • You'll get your count updated automatically.

  • Then you take slice 0 and slice 1 and [INAUDIBLE]..

  • And you invoke a function.

  • You're not interested in slice 0.

  • You'll want only to get the second value.

  • So just keep it.

  • And you're trying to print it.

  • So at a GraphDef level and FunctionDef,

  • it will look something like this.

  • It will have a good slice function

  • that takes three sources, a, b, and c.

  • It will have an increment for the counter.

  • It will read the variants of a, b, take the slices.

  • And return slice 0 and slice 1.

  • And it will have a control [INAUDIBLE] increment

  • automatically added by automatic [INAUDIBLE] tracking.

  • So slices don't have any control dependencies

  • because slices, they do not depend on variable c.

  • They depend on different variables.

  • And then you have a function call,

  • a partition call to the get slice a and b.

  • And you don't need the first slice.

  • You take the second one by a function called output 1.

  • And you're trying to fetch it, or print it, or whatever.

  • And the problem right now, that this code,

  • we will fail because the function semantics [INAUDIBLE]

  • flow currently is strict with regards

  • to its inputs and outputs.

  • So before function return, function can return,

  • it must finish the execution of all the output parameters.

  • And before function can start executing,