And I just want to show you how simple, smooth, and effective

running open source, big data software

can be on Google Cloud Platform.

Now, let's just begin by considering

what we might expect out of any modern programming language.

It seems it's not quite enough to be turning complete.

But we probably at least expect some standard libraries

for common things like say, sorting a one megabyte list

in-memory.

Well, nowadays, it seems like everybody

has a lot more data lying around.

So what if it could be just as easy to sort a terabyte of data

using hundreds or even thousands of machines?

More recently programming languages

have had to incorporate much stronger support for things

like threads and synchronization as we all moved on

to multi-processor, multi-core systems.

Nowadays, it's pretty easy to distribute work

onto hundreds of threads.

What if it could be just as easy to distribute work

onto hundreds of machines?

Nowadays, it's also pretty common

to find some fairly sophisticated packages

for all sorts of things like cryptography, number

theory, and graphics in our standard libraries.

What if we could also have state of the art distributed machine

learning packages as part of our standard libraries?

Well, if you've been watching the code snippets over here

the last few slides, you may have

noticed that none of what I just said is hypothetical.

It's all already possible right now, today.

Which is fortunate because just as we spent the last decade

finally getting the hang of programming

for multi-threaded systems on a single machine, nowadays

it seems more and more common to find yourself developing

for a massively distributed system of hundreds of machines.

And with these changes in the landscape,

it's pretty easy to imagine that big data could

be a perfect match for a cloud.

All the key strengths of cloud like on-demand provisioning,

seamless scaling, the separation of storage from computation,

and of course, just having such easy access

to such a wide variety of services and tools.

These all contribute to opening up

a whole new universe of possibilities

with countless ways to assemble these building

blocks into something new and unique.

And through this shared evolution

of big data and distributed computing,

big data has become a very core ingredient of innovation

for companies big and small, old and new alike.

Take, for example, how streaming media companies like Netflix,

Pandora, and Last.fm have contributed

to changing the way that consumers

expect content to be presented.

They've applied large scale distributed machine learning

techniques to power these personalized recommendation

engines.

Millions of users around the world

now expect tailor-made entertainment as the norm.

As another example, just taking a look at our Google's roots

in web search and along side a multitude of other search

engines, news aggregators, social media,

and e-commerce sites, we've all long

grappled with applying big data technologies just

to tackle the sheer magnitude of ever-growing web content.

Users now expect to find a needle

in a haystack hundreds of thousands of times,

every second of every day.

And amidst this rise of big data,

it's not that the data or the tools

have suddenly become magical.

All these innovations ultimately come from developers

just like you, always finding new ways

to put it all together so that big data can still

mean something completely different to each person

or entrepreneur.

On Google Cloud Platform, through a combination

of cloud services and a wealth of open source technologies

like Apache Hadoop and Apache spark,

you'll have everything you need to focus

on making a real impact instead of having to worry about all

the grungy little details just to get started.

Now for an idea of what this all might look like,

let's follow a data set in the cloud on its journey

through a series of processing and analytic steps.

And we'll watch as it undergoes its transformation

from raw data into crucial insights.

Now what I have here is a set of CSV

files from the Center for Disease Control containing

summary birth data for the United

States between 1969 and 2008.

These files are just sitting here in Google Cloud Storage.

And, as you can see, it's pretty easy to peek here and there

just using gsutil cat.

Now suppose we want to answer some questions by analyzing

this data like, for example, whether there's

a correlation between cigarette usage and birth weight.

At more than 100 million rows, it

turns out we have at least 100 times too much data

to fit into any normal spreadsheet program.

So that means it's time to bring out the heavy machinery.

With our command line tool, bdutil, and just

these two simple commands, you can have your very own 100 VM

cluster fully loaded up with Apache Hadoop, Spark,

and Shark, ready to go just five minutes or so after kicking it

off.

Once it's done bdutil will print out a command for SSHing

into your cluster.

Once we're logged into the cluster,

one easy way to get started here is just

to spin up a Shark shell.

We just type shark and this prompt will appear.

Now, what we'll be doing here is creating

what's known as an external table.

So we'll just provide this location parameter to Shark

and point it at our existing files in Google Cloud Storage.

Shark will go in and list all the files

that match that location.

And we can immediately start querying those files in place,

treating them just like a SQL database.

For example, once we have this table loaded,

we can select individual columns and sort on other columns

to take a peek at the data.

We can also use some handy built-in functions

to calculate some basic statistics

like averages and correlations.

Now one thing I've noticed about this kind of manual analytics,

is that while it's a great way to answer some basic questions,

it's an even better way to discover new questions.

For instance, over here we've found a possible correlation

between cigarette usage and lower birth weights.

So could we, perhaps, apply what we

found to build a prediction engine for underweight births?

And what about other factors like the parent's age

or alcohol consumption?

Now, this kind of chain reaction of answers leading

to new questions is part of the power of big data analytics.

With the right tools at hand, then every step along the way

you tend to find new paths and new possibilities.

In our case, let's go ahead and try out

that idea of building a prediction

engine for underweight births.

Now, to do this we're going to need some distributed

machine learning tools.

Traditionally, building a prediction model on 100 node

cluster might have meant years of study

and maybe even getting a Ph.D. Luckily for us,

through the combined efforts of the open source community

and, in this case, especially UC Berkeley's AMPLab,

Apache Spark already comes pre-equipped with a state

of the art distributed machine learning library called MLlib.

So our Spark cluster already has everything

we need to get started building our prediction engine.

The first thing we'll need to do here

is just to extract some of these columns

as numerical feature vectors to use as part of our model.

And there's a few ways to do this.

But since we already have a Shark prompt open,

we'll just go ahead and use a select statement

to create our new data set.

We'll start out with a rough rule of thumb here, just

defining underweight as being anything less than 5.5 pounds.

And we can just select a few of these other columns

that we want to use as part of our feature vectors.

We'll use this WHERE clause to limit and sanitize our data.

And we could also just chop off the data

from the year 2008, just for now,

to put in a separate location, so that we

can use that as our test data, separate from our training

data.

Now, since we're doing all this as a single create table

as a select query statement, all we have to do

is provide a location parameter and tell Shark

where to put these new files once it's created them.

We can also go ahead and run that second query on the data

from 2008, so that we'll have our test

data available in a separate location, ready to go later.

Sure enough, after running these queries,

we'll find a bunch of new files have appeared in Google Cloud

Storage, which we can look at just using gsutil or Hadoop FS,

for example.

Now, with all our data already prepared, all we have to do

is spin up a Spark prompt so that we have access to MLlib.

It just so happens we can pretty much copy and paste

the entire MLlib Getting Started example for support vector

machines, here.

And we'll just make a few minor modifications that

point it our training data here, and plug that into Spark.

Spark will go ahead and kick off the distributed job,

entering it over the state at 600 times

to train a brand new SVM model.

Now, that will take a couple minutes.

And once that's done, we'll have a fully trained model

ready to go to make predictions.

Here, for example, we can just plug

in a few examples of underweight and non-underweight predictions

using data from our real data set.

We can also go in and load that separate data from 2008

that we saved separately, and rerun our model against it

to make sure that our error fraction is still comparable.

Now, taking a look at everything we've done here,

I'll bet we've raised more questions than we've answered,

and probably planted more new ideas

than we've actually implemented.

For instance, we could try to extend this model

to predict other measures of health.

Or maybe we can apply the same principles

but to something other than obstetrics.

We could also explore a whole wealth

of other possible machine learning tools

coming out of MLlib.

Indeed, as we dive deeper into any problem

we'll usually find that the possibilities are pretty much

limitless.

Now, sadly, since we don't quite have

limitless time in this video session,

we'll have to come to an end of this leg of the journey.

Now everything you've seen here is only a tiny peek

into the ever ongoing voyage of data

through ever-growing stacks of big data analytics

technologies.

Hopefully, with the help of Google Cloud Platform,

your discoveries can reach farther and spread faster

by always having the right tool for the right job,

no matter what you might come across.

Thanks for tuning in.

I'm Dennis Huo.

And if you want to find out more,

come visit us at developers.google.com/hadoop.