I'm Sujith.

I lead a few machine learning teams in Google AI.

We work a lot on--

how do you do deep networks and build machine learning systems

that scale on the cloud with minimal supervision?

We work on language understanding, computer vision,

multi-modal applications.

But also we do things on the edge.

That means, how do you take all these algorithms

and fit it to compute and memory-constrained devices

on the edge?

And I'm here today with my colleague.

DA-CHENG JUAN: Hi, I'm Da-Cheng.

And I'm working with Sujith on neural structured learning

and all the related topics.

SUJITH RAVI: So let's get started.

You guys have the honor of being here for the last session

of the last day.

So kudos and bravo--

you're really dedicated.

So let us begin.

So we are very excited to talk to you

about neural structured learning, which

is a new framework in TensorFlow that

allows you to train neural networks

with structured signals.

But first, let's go over some basics.

If you are in this room and you know about deep learning

and you care deeply about deep learning,

you know how typical neural networks work.

So if you want to take, for example, a neural network

and train it to recognize images and distinguish

between concepts like cats and dogs, what would you do?

You feed images like this on the left side, which

looks like a dog, and give the label "dog,"

and feed it to the network.

And the process by which it works

is you adjust weights in the network such

that the network learns to distinguish and discriminate

between different concepts and correctly tag the image

and convert the pixels to a category.

This is all great.

All of you in this room probably have built a network.

How many of you have actually built a neural network?

Great!

And this is the last session, last day--

but still, we're very happy that you're all with me here.

So it's all great.

We have a lot of fancy algorithms,

very fancy networks.

What is the one core ingredient that we

need when we build a network?

So almost majority of the applications that we work on,

we require label data, annotated data.

So it's not one image that you're feeding to this network.

You're actually taking a bunch of images paired

with their labels, cats and dogs in this case,

but of course it could be whatever,

depending on the application.

But we feed it thousands or hundreds

of thousands or even millions of examples into the network

to train a good classifier, right?

Today, we're going to introduce neural structured learning,

which is a framework.

We're happy to say it's support an in TensorFlow 2.0 and Keras.

And it allows you to train better and more robust

neural networks by leveraging structure in the data.

So the core idea behind this framework

is that we're going to take neural networks

and feed it, in addition to feature

inputs, structured signals.

So think of the abstract image that I showed you earlier.

Now in addition to these images paired with labels,

you're going to feed it connections or relationships

between the samples themselves.

I will get to what these relationships might mean.

But you have these structured signals and the labels.

And you feed both into the network.

You might ask, what do you mean by structure, right?

Structure is everywhere.

In the example that I showed you earlier,

if you look at images--

just take a look at this graph here.

The images that are connected via edges in this picture

here basically represent that there's some visual similarity

between these.

So it is actually pretty easy to construct structured signals

from day-to-day sources of data.

So in the case of images here, it's visual similarity.

But you could think of--

what if you tag your images and created

albums that represent some specific concepts?

So everything within an album or a photo album

has some sort of a connection or interaction or relationship

between them.

So that represents another type of structure.

It's not just for images.

We can go to more advanced or completely different

applications, like, if you want to take scientific publications

or news articles and you want to tag them with their topic--

one simple thing.

Take biomedical literature.

All the papers that are published,

whether it's Nature or any of the conferences,

they have references and citations to other papers.

That represents another type of structure or link.

So these are the kind of structures

that we're talking about here, relationships

that are exhibited or modeled between different types

of objects.

In the natural language space, this occurs everywhere.

If you're talking about doing Search,

everybody has heard of Knowledge Graph, which

is a rich source of information, which captures relationships

between entities.

So if I talk about the concept Paris and France,

the relationship is one is the capital of the other.

So these sort of relationships--

it's not typical to capture and feed them

into a neural network.

But these are the kind of relationships

which are already existing in day-to-day data sources.

So why not leverage them?

So that is what we try to do with the neural structured

learning.

And the key advantages--

before we talk about what it does and how we do it,

why do you even want to care about it?

What is the benefit?

So one of them is, as I mentioned earlier,

it allows you to take this structure

and use it to train neural networks with less

labeled data.

And that's the costly process, right?

So every application that you want to train,

if you had to collect a lot of rich annotated data

at scale for millions of examples,

it's going to be a tedious task.

Instead, if you're able to use a framework,

like neural structured learning, that automatically captures

a data structure and relationship,

with minimal supervision you're able to train

classifiers or prediction systems with the same accuracy.

That would be a huge boon.

Who wouldn't want that, right?

That's one type of a benefit.

Another one is that, typically, when you deploy these systems

in practice, in real-world applications,

you want the systems or networks to be robust.

That means, you train the ones you don't want--

if the input distribution changes or data suddenly

changes or somebody corrupts the images with adversarial

attacks--

suddenly the network to flip the predictions and go bonkers.

So this is another benefit where,

if you use neural structured learning,

you can actually improve the quality of your network

and also the robustness of the network.

So let me dive a little deeper and give

you a little more insight into the first scenario.

Take document classification as an example.

So I'll give you, probably, some example that you probably

have at your home.

Imagine you have a catalog or a library of books in your home.

And these are digitized content.

And you want to categorize them and neatly arranged them

into specific topics or categories.

Now one person might want to categorize them

based on the genre.

A different person might say, oh,

I want it to belong to the same period.

A third person might say, oh, I want

to capture the books that have the same kind of content

or phrases or words that appear by the same author

or based on some certain aspect.

Like, there is a particular plot twist

that you have that is captured in different books.

And I want to arrange them based on that.

So you can think of--

not everybody's needs are the same.

So are you going to collect and annotate

enough label data for each of those tasks

to create a network with very high accuracy,

distinguish and classify them into these genres?

Probably not.

So on the one hand, you have plenty of data.

The raw book content is available to you.

Or raw news articles are available to you.

There's plenty of raw text available to you.

But it's hard to construct this labeled annotated data.

So this is where Neural Structured Learning or NSL--

going back to the previous example,

we can model the relationships between these different inputs

or samples, using structure.

Again, I will tell you what the structure means.

But pair it with a few label examples

and then train a network that is almost as

good as the network that is trained

on millions of examples.

So imagine you could, for your application, only use 5% or 10%

of the label data and train as good

a classifier or prediction system.

So that's what we're trying to do

and help you do with neural structured learning.

And I said, structure-- how do you

come up with this structure?

Or how do you do this for document classification?

I'm going to give you a forward reference

here that Da-Cheng is going to talk a little bit more

about as well.

So there is a hands-on tutorial for the exact example

that it told you, document classification with a graph.

And you can go to the Neural Structured Learning TensorFlow

website and try this out for yourself,

all within just a few lines of code.

All you need to do is construct the data and the format.

And with a few lines of code, you

should be able to run the system end-to-end.

Switching to the second scenario,

it's great to have high-quality models.

We have seen that neural networks have the capacity

to train really, really good quality models,

especially as you increase the number of parameters.

But robustness is an important concept and, actually,

an important criteria when we deploy this

to real-world scenarios.

For example, if you have an image recognition system

and suddenly the network flips its prediction,

because the images are corrupted,

this is not a good thing.

And this is not just for image classification.

It could be with text or any kind of sensor data.

Here is an example.

Take the image on the left.

What do you think it is?

It's a panda.

Take the image on the right.

What do you think it is?

Anyone who disagrees it's a panda?

Basically, your neural network is

saying that both of these images are pandas.

But that's not what happens if you actually train

a neural network, like, for example, resonance or whatever,

the state of the [INAUDIBLE] network

and you try to apply it to the two images.

The first one would be correctly recognized as a panda.

The second one would be recognized

by a completely different concept,

in this case, a gibbon.

And the reason it happens is, if you zoom in really close,

the second image is actually an adversarial example.

It's actually created by adding some noise

to the original image.

But these are so tiny.

It's very imperceivable for the human eye.