ZAK STONE: Thank you very much.

I'm delighted to be here today to talk to you about some

of the fantastic TensorFlow Research Cloud projects

that we've seen around the world and to invite

you to start your own.

Whether you're here in the room, watching on the livestream,

or online watching this afterwards, any of you

are welcome to get involved with TFRC.

Just very briefly, since I'm sure you've

heard this all today, the context

is this massive improvement in computational capabilities

driven by deep learning.

So deep learning, and specifically

these deep neural networks, are enabling many new applications

that are exciting in all sorts of ways,

touching all kinds of different data,

ranging from images, to speech, to text, even full scenes.

And the challenge that many of you are probably grappling with

is that these new capabilities come with profound increases

in compute requirements.

A while back, OpenAI did a study where

they measured the total amount of compute required

to train some of these famous machine learning models

over the past several years.

And the important thing to notice about this plot

is that it's actually a log scale on the compute axis.

So there are tremendous increases in the total amount

of compute required to train these state-of-the-art deep

learning models over time.

And there's this consistent trend up and to the right,

that these new capabilities are being unlocked

by the additional compute power, as well as lots of hard work

by many researchers all around the world

in this open community.

So unfortunately, these tremendous demands

for compute to meet these new opportunities opened up

by deep learning are coming to us

just as Moore's law is ending.

We've benefited for decades upon decades

in consistent increases in single-threaded CPU

performance.

But all of a sudden, now we're down to maybe 3% per year.

Who knows?

There could always be a breakthrough.

But we're not expecting extraordinary year-upon-year

gains from single-threaded performance,

as we've enjoyed in the past.

So in response to that, we believe

that specialized hardware for machine learning

is the path forward for major performance wins, cost savings,

and new breakthroughs across all these research

domains that I mentioned earlier.

Now, at Google, we've developed a family

of special-purpose machine learning

accelerators called Cloud TPUs.

And we're on our third generation now,

and two of these generations are available in the cloud--

the second and the third generation.

Just to give you a brief overview of the hardware

that I'm going to be talking about for the rest

of the session, we have these individual devices here--

Cloud TPU v2 and v3.

And as you can see, we've made tremendous progress generation

over generation--

180 teraflops to 420 teraflops.

We've also increased the memory from 64 gigabytes

of high bandwidth memory to 128, which

matters a lot if you care about these cutting-edge natural

language processing models, like BERT, or XLNet, or GPT-2.

But the most important thing about Cloud TPUs

isn't just these individual devices,

which are the boards that you see here

with the 40 TPU chips connected to a CPU host that's not shown.

It's the fact that these devices are designed

to be connected together into multi-rack machine learning

supercomputers that let you scale much further,

and program the whole supercomputer

across these many racks as if it were a single machine.

Now, on the top here, you can see the cloud TPU v2 pod

spanning four racks.

The TPUs are in those two central columns,

and the CPUs are on the outside.

That machine got us to 11 and 1/2 petaflops,

which you can also subdivide every which way, as you wish.

And the TPU chips, in particular,

are connected by this 2-D toroidal mesh

network, that enables ultra fast communication.

It's much faster than standard data center networking.

That's a big factor in performance,

especially if you care about things like model parallelism

or spatial partitioning.

But now, with the Cloud TPU v3 Pod,

which is actually liquid-cooled, the picture

wasn't big enough to hold all the racks.

It spans eight racks out to the side,

and it gets you up over 100 petaflops

if you're using the entire machine simultaneously.

On a raw op by op basis, that's competitive with the largest

supercomputers in the world.

Although, these TPU supercomputers

use lower precision, which is appropriate for deep learning.

Now, I've mentioned performance.

I just wanted to quantify that briefly.

In the most recent MLPerf training version 0.6

competition, Cloud TPUs were able to outperform

on premise infrastructure.

What you can see here is the TPU results

in blue compared with the largest on-premise cluster

results that were submitted to the MLPerf competition.

And in three of the five categories that we entered,

the Cloud TPUs delivered the best top line

results, including 84% increases over the next entry in machine

translation, which is based on transformer and object

detection, which was an SSD architecture.

Now, obviously, these numbers are evolving all the time.

There's tremendous investment and progress in the field.

But I just wanted to assure you that these TPUs can really

deliver when it comes to high performance at scale.

But today we're here to talk about research and expanding

access to this tremendous computing power

to enable researchers all over the world to benefit from it,

and explore the machine learning frontier,

make their own contributions to expand it.

In order to increase access to cutting-edge machine learning

compute, we're thrilled to have been able to create

the TensorFlow Research Cloud, in order to accelerate open

machine learning research, and hopefully to drive this

feedback cycle, where more people than ever before have

access state-of-the-art tools.

They have new breakthroughs, they

published papers, and blog posts, and open source code,

and give talks, and share the results with others.

That helps even more people gain access to the frontier

and benefit from it.

So we're trying to drive this positive feedback loop.

And so, as part of that, we've actually made well

over 1,000 of these Cloud TPU devices available for free

to support this open machine learning research.

If you're interested in learning more right now,

you can go to g.co/tfrc.

I'll also have more information at the end of the talk.

This pool of compute--

the TFRC cluster-- involves not just the original

TPU v2 devices that we included, but we've recently added

some of the v3 devices of the latest generation,

if you're really pushing the limits.

And there's the potential for Cloud TPU pod access.

If you've gone as far as you can with these individual devices,

please email us, let us know, and we'll

do our best to get you some access to TPU pods.

The underlying motivation of all this

is a simple observation, which is that talent is equally

distributed throughout the world, but opportunity is not.

And we're trying to change that balance,

to make more opportunities available to talented people

all around the world, wherever they might be.

So we've had tremendous interest in the TFRC program so far.

More than 26,000 people have contacted

us interested in TFRC, and we're thrilled

that we've already been able to onboard

more than 1,250 researchers.

And we're adding more researchers all the time,

so if you haven't heard from us yet, please ping us again.

We really want to support you with TFRC.

The feedback loop is just starting to turn,

but already I'm happy to announce

that more than 30 papers in the academic community

have been enabled by TFRC, and many of these researchers

tell us that without the TFRC compute,

they couldn't possibly have afforded

to carry out this research.

So I feel like, in a small way, we

grabbed the lever of progress here,

and we have tipped it slightly upward.

So the whole field is moving just a little bit faster,

and we really thank you all for being part of that.

I'm most excited though to share some of the stories

directly, of the individual researchers and the projects

that they've been carrying out on the TFRC Cloud TPUs.

Now these researchers, they come from all over the world.

I only have time to highlight four projects today,

but the fantastic thing is that three of these researchers

have been able to come and travel here

to be with us in person.

So you'll get to hear about their projects

in their own words.

We'll start with Victor Dibia, here in the upper left.

Welcome, Victor.

Come on up.

[APPLAUSE]

VICTOR DIBIA: Hi.

Hello, everyone.

Really excited to be here.

My name is Victor Dibia.

I'm originally from Nigeria.

And currently, I'm a research engineer

with Cloudera Fast Forward Labs in Brooklyn, New York.

And so, about a year ago, I got really fascinated

about this whole area of the intersection of art and AI.

And given my background and my interest

in human-computer interaction and applied artificial

intelligence, it was something I really wanted to do.

Right about the time, I got the opportunity

to have access to TFRC, and today, I'm

going to talk to you about the results of those experiments,

and some of the research results I had working on this.

And so, why did I work on this project?

As a little kid growing up in eastern Nigeria,

my extended family and I would travel to our village

once a year.

And one of the interesting and captivating part of those trips

was something called the Eastern Masquerade dances of Africa.

And so, what would happen is that there

is these dancers with these complex, elaborate masks.

And as a kid, I was really fascinated,

and so, this project was a way to bridge

my interest in technology arts.

And as a research engineer too, I

could also express my identity through a project like this.

In addition to this, there's this growing area

of AI inspired art, or AI generated art.

But one thing you'll notice in that space

is that most of the data sets that are used

for this sort of explorations are mainly

classical European art--

Rembrandt, Picasso.

And so, a project like this is a way

to diversify the conversations in that area.

And then, finally, the researchers

working in the generative model domain, one of the goals

here is to also contribute to more complex data sets,

compared to things like faces or retail images.

And it can be a really interesting way

to benchmark some of the new generated models

that are being researched today.

So what did I do?

So like all of us know, the best results

or most of the effort from machine learning projects

comes from the data collection phase.

So I started out collecting images.

I curated about 20,000 images, and then

resolved that down to about 9,300 high-quality images.

And at this point, I was ready to train my model.

And so, the beautiful thing is that the TensorFlow team