clustering classification data pre-processing

big data

It's hard to go anywhere now without hearing about AI and machine learning and data data, particularly

It's everywhere research

We've suggested that every two years we generate more data than ever existed before

So the amount of data is doubling every two years now, that isn't absolutely am, you know astronomical amount of data

but the thing is of course that

This data doesn't necessarily mean anything the fact you can create tables of data

But unless you understand what's in them and what they mean, you haven't got any knowledge, right?

So there's a distinction between having data and having knowledge. So all very well saying yes as a species

We're producing a huge amount of data

But actually a lot of it doesn't get used a lot of it sits there on a hard disk

Waiting for someone to look at it and that's kind of what we're talking about here if we want to extract knowledge from data

we're going to need some tools and processes to do this in a formal way and that's that's what data science is, right and

Things like machine learning and AI have a place within it

So perhaps if you do this for your job, then data analysis is going to be useful for you

Maybe your company's generating data and you want to analyze this data?

But on the other hand, perhaps you're just a consumer and companies are using data on you. They're generating data on you

And actually they're profiting from data on you. These are sometimes life-changing decisions that are being made on your data

And so it's empowering to know how this process works and I'm a very simple example

Which you might even do yourself suppose you go online to book some flights for a holiday

And then you decide that actually two flights via an intermediate Airport is cheaper than a single flight, right?

You're doing data analysis

Say you're taking lots of different data sources and working out the optimal route and this of course happens automatically as well

Depending on the flight website that you're using. All right, so this kind of stuff you're already doing it

It's just a case of trying to formalize this process. So what do any of the things I listed at the beginning mean?

Well one problem is that everyone's definitions differ slightly

But also I think that a lot of these terms are used completely interchangeably AI is the classic example

So AI is everywhere right talk

You can't buy a product without it having been having AI added to it a lot of the time you see AI

We're actually talking about machine learning

so machine learning is the idea that we're

Training a machine to perform a task without explicitly

programming it to do so. A good example of AI that isn't machine learning would be lets say a mouse in a maze where all

You're doing is telling it to turn left or right at random not learning anything

It doesn't understand what the maze is but it will eventually get to the end right that's a kind of rudimentary artificial intelligence

That doesn't involve learning anything

Machine learning is about not giving it

Conditions not saying if you're here turn left if you're here turn, right

It's just giving it examples and hoping it will learn to perform most tasks itself, right?

So machine learning is a subset of AI but they shouldn't be used interchangeably if we're using machine learning often

What we'll do is we train it based on samples of data

So we'll have some existing data set that we're trying to train on and we're trying to use the machine learning to either tease out

information or make predictions on this data

The problem is that not all data is sort of made equal some of its noisy and messy

Maybe we don't know what it is and don't know whether we can apply a certain technique to it

Right. And so we need to clean this data up. We need to take this data understand what it is and extract some knowledge

So that we can then apply these AI or machine learning techniques to it

So this combination of things that can take data and prepare it in a way that we can then use it or understand it

That's data science

There are quite a few ways we could do this data analysis right throughout this course

We could use R, we could use Python, we could use MATLAB. They all have their pros and cons

We're gonna use R because it's free and it's really good for statistical analysis

It's got loads of great libraries

If you're really familiar with Python, then maybe that's what you want to start with for this kind of stuff

But we know we're going to be working with R

We have our script area here where we can write scripts and run scripts. You can save them and then come back to them later

Console where we're going to be putting in, you know specific commands

we have our environment which is where all our

Variables and our data is held and we can look at them there and then we have plots any plots of which you can do

quite a lot of different plots in R, very versatile. That's going to appear down here

Okay, so you've probably got everything you need to get started with data analysis. In my opinion

The best way to get into R is just to kind of have a go

So it's going to look at a few of the most obvious things that it does it has

A little bit of a learning curve only because it's syntax is slightly unusual

If you can program you'll be fine

but even if not

you should get there pretty quickly. Most of the time in R we'll be using either matrices or vectors or

Which are kind of a special case of matrices or maybe data frames data frames a really nice aspect of R which you can kind

Of think of like a table that you might have in in Excel, except you've also got headings for your columns

so let's have a look at some of these things and just a few of the things we can do with them before we perhaps

Go into a little bit more detail in other videos

so for example

We might look at our variable X which I've created and X is a sequence going from 0 all the way up to a few

multiples of Pi which I used to create this plot

That was only one line of code that produced that and I've used that to create my plot by essentially saying y equals sine X

And then just simply plotting that if you wanna get a little bit more complicated we can start looking at matrix data

So I created a CSV file with a Gaussian function in it. So essentially a two dimensional array of

Values that get bigger in the center very straightforward

the CSV file is essentially a text file with

commas separating those values very easy to read and write these out of Excel and other

packages and so they're off you'll often find data is passed around in this way at least

Moderately sized data, if it isn't too, you know to it too huge. I can load this in using my read CSV function

So I can say name data

Now the arrow operator is essentially equivalent in R for the assignment operators or equals equals will often work

But I tend to try and use this one. So namedata

I'm going to assign read dot CSV and the file is going to be norm dot CSV

And I've got no header for this file. So I don't want it to use the top row for the labels

So I'm going to say header equals

false and that's loaded in namedata and we can have a look so I'm gonna click on namedata here and if we click

On it you can see we've got the rows and the columns of our data in here

We can look at individual elements in this array so we can say data at position three four

right

And that's going to be the third row down and the fourth value across we can also leave one empty and just have an entire

row or

Conversely an entire column like this and so it's very easy to take ranges of values

You've got a huge table of data selecting certain columns looking at certain columns plotting certain columns

This is one of the reasons why R is very popular quite often when you're looking at data

We'll actually be looking at something called a data frame. Now a data frame. I've got a load one up is simply a

In essence a table of values, but it will have to be the same type

So in an array, normally they'll all be floats or they'll all be integers. In a data frame, there can be different things

So you could have first and last name next to age. For example

So I've just created a tiny little CSV file with some random people in it. So let's load this up

So I'm going to say namedata

assign read CSV

names dot

CSV and if I look at name data, you can see that it's got three columns

it's got first name surname and age and

Five rows and there's five people in this dataset and then you can do just like I did before but now we can also index

By the names of these columns so I could say I want all of the first names for example so I can say namedata

dollar

first-name and I can see

All the different first names so you can start to look at this data set and more in more detail, obviously

This isn't absolute tiny data set but you get the idea you could also look at individual instances

So we could say name data and I want just the second row for example name data the second row

There we go, Bill Jones and he's 18 years old as we move through these videos

It's going to be very common for us to load in

Datasets like this in this format and then start to process them based on these data frames. So perhaps an example, right?

so, so let's imagine you're an online retailer and someone comes into your shop and buy some things and maybe they you

Trying to understand what it is what they do so that you can let's say send them emails to try and get them to buy

More products or show them recommended products and things like this

So you want to try and build up a pattern of their behavior, right?

And all you've got is what they click on what they add to their basket and what they buy, right?

So you've learned that they're looking at these kinds of items and they look at these ones regularly

And then sometimes they just buy something completely random seemingly, and that goes in their basket and gets bought straight away

Maybe it's a present right? So maybe it's not tied to them as a person

So you're taking all of this data all of these purchases all of these?

Products are they're looking at and you're turning this into a kind of picture of this person and you're clustering that person in with other

consumers that bought similar things and trying to predict what they want to buy next, right?

And that's when you send them an email say you should look at this one because this one's really good and you didn't buy it

Last time but you'll definitely want to buy it this time. So we've got some data we want to extract some knowledge

What's the first thing we do?

well

We have to start to look at it and try and tease out some kind of information

Right or analyze this data the data analysis is the idea of using statistical measures to try and work out what's going on

This is kind of a cycle. We're going to analyze the data

So we're going to do a data analysis and perhaps sometimes just using statistics to analyze the data isn't enough

You can't really learn everything about it

Yes, you can learn, you know, mathematically how it works, but you might not understand about what it all means

So visualizing the data can be really helpful. So what we'll also do is we'll visualize the data

Visualization so that's going to be charting it plotting it trying to work out

trends and

Links between different variables and things like this and these are kind of being back and forth

Right, you could do both of these things numerous times and work out what we've got, right?

So you're gonna do something like this. And then what we're going to do is we're going to pre-process the data

Often you'll be finding your recording much more data than you actually need. Right. This is certainly true of an online shop

I'm going to be looking at a lot of products

But I don't end up buying and I was never really going to buy I know maybe a pipe dream and they've got a sort

Of weed out this information to work out what it is that they might actually better convince me to buy right?

So this is going to you going to preprocess data and remove a nonsense and drill right down to the stuff that's really useful

So this is pre-processing and this is going to be a kind of cycle of analysis and visualization

and

Pre-processing and we can repeat these things and then we can really drill down and whittle down our data into the most usable sort of

Core of knowledge that we can

And get the most out of it. Now it may be that just analysing the data is enough, right?

You've now sort of you've obtained some knowledge

You kind of understand what the trends are and maybe that was all you wanted to do. That's sometimes the case

Maybe actually what we want to do is take things a little bit further

We're going to use machine learning or modeling to try and model this system and predict what's going to happen next?

So for example in the case of an online shop

We might want to start predicting what people are going to buy next and if we can do that

That's when we can send out these emails or flag things in their recommended items and get many more sales as an example

Let's imagine that someone has spent a lot of time looking at DIY tools right. I've you know recently moved house

I spent a lot of time doing DIY and I'm always trying to buy new tools because it just seems like a good idea

So, you know, maybe I buy a certain kind of saw and then you know a few months later. They're starting to recommend me

a slightly different kind of saw