Last week I ordered a pair of gold lame pants with DFTBAQ embroidered on them.

The delivery guy said they could come by the next day at exactly 11am on the dot!

Just kidding. That never happens.

Instead of an exact time, the pants guy gave me a range of times...he said they'd be

there sometime between 8am and 2pm.

A lot of anticipation…

We've focused a lot on point estimates, like the mean, which are our best guesses,

but we can give ourselves a little more wiggle room.

Let's talk about Confidence Intervals.

INTRO

It's useful to give pregnant mothers a “due date” when their children will most likely be born.

But it might be more accurate to say that doctors expect the baby to come around the

due date, not exactly on it.

And...when pollsters claim that a candidate will get around 30% of the vote, plus or minus 2%.

We can represent the “around” part with a confidence interval.

You may have seen the term “confidence interval” paired with a percentage like 95%.

A “confidence interval” is an estimated range of values that seem reasonable based

on what we've observed.

It's center is still the sample mean, but we've got some room on either side for our uncertainty.

So when the delivery guy says my pants are coming between 8 and 2--he's reflecting

his uncertainty...the very LARGE frustrating uncertainty, about when he'll be there.

For example, a dentist thinks the mean number of cavities the average person has in a 5

year span is greater than 1 and wants to calculate a 95% CI to see if there's evidence that he's right.

He rounds up a random sample of 100 patients from around the country, and finds that this

group has a mean of 3 cavities with a standard deviation of 0.5 cavities.

The way we choose that confidence range is related to the distribution of sample means.

The dentist's estimate of the sampling distribution looks like this:

And instead of grabbing just the mean, the dentist can include a range of the most common

95% of the sample means that we expect from this estimate of the distribution of sample means.

So now we have a 95% confidence interval from 2.902 to 3.098 cavities.

Giving a range of numbers instead of just an estimate for the mean better represents

the fact that there's some uncertainty and variation when we estimate population parameters--like

the mean, proportion, or regression slope--from a sample.

The interpretation of this confidence interval is a bit more complex.

To understand what a confidence interval really is, we have to ask ourselves “what if?”.

If the dentist's sample was taken again, we wouldn't expect that the mean and standard

deviation of cavities would be exactly 3 and 0.5.

They'd probably be a little different.

Which means that our 95% confidence interval would be different than the one we got before.

And if we did it 100 more times with the same sample size, we'd get 100 slightly different

confidence intervals.

The 95% in a 95% confidence interval tells us that if we calculated a confidence interval

from 100 different samples, about 95 of them would contain the true population mean.

Our “confidence” is in the fact that the procedure of calculating this confidence interval

will only exclude the population mean 5% of the time.

That definition implies that it's possible that the confidence interval that we created

doesn't include the true population mean.

We have no way of knowing for sure.

But the confidence intervals usually contain the true population mean.

Now that we know what a confidence interval is, it might be useful to calculate it.

A 95% CI is the range that contains the middle 95% of the values of our estimated sampling distribution.

And to get that range, we can use a z-score.

A z-score tells us the distance between the mean of a distribution and a data point in

standard deviations.

Previously, we've used z-scores to help us find percentiles.

And we want the middle 95% of the data.

So we want our cutoffs to be at the 2.5th percentile and the 97.5th percentile so that

95% of the values are within our range, and 5%--2.5% on either side--are not.

To calculate the 95% confidence interval for a sample of 49 chocolate cakes with a mean

of 3,000 calories and a standard deviation of 500 calories, we can use a z-score of 1.96

(which we got from a table) to calculate the 97.5th percentile, and a z-score of -1.96

to calculate the 2.5th percentile.

But we need to turn our z-scores back into calorie values.

To do so, we multiply by the standard error, 71.4 calories and add the mean of 3,000 calories

to get the 95% confidence interval for our sample.

We think it's likely that the real population mean for number of calories in a chocolate

cake is in that range, though we're not sure.

What we can have confidence in, is that if we're in a situation where we're constantly

taking samples like this and we assume that the true mean is inside of every Confidence

Interval, we'll only be wrong 5% of the time.

For example, a gummy worm factory periodically checks whether their bagging machines are

calibrated correctly.

So each week, they take a sample of 100 bags of gummy worms, measure the mean weight and

standard deviation, and calculate a 95% confidence interval.

They use the Confidence interval to make a decision about whether to pay an expensive

repair man to come repair the gummy worm bagging machine.

They want their bags of gummy worms to have around 10oz of gummy treats, and decide that

as long as the confidence interval contains 10oz--their ideal weight--they'll assume

their machine is fine.

Decisions based on their confidence intervals will lead them to call an unnecessary repairman

only 5% of the time.

Many researchers use confidence intervals to see if they contain a certain value of interest.

A researcher may want to know if say a certain number of calories in cake is plausible.

If the sampled value were to fall within their CI it would seem possible, but it's not

possible to rule out even if it's outside the interval.

Because you don't know if you got the 95% of CI's that contain the true mean or the

5% that don't.

You don't always need to use a confidence interval of 95%, we can calculate other confidence intervals too.

You can calculate a 99% confidence interval, or really any percentage confidence interval.

But if you try to calculate a 100% confidence interval, it'll always be negative infinity

to positive infinity, which just shows that the larger you want your confidence percentage

to be, the wider your interval will be.

You can be more hopeful that your confidence interval contains the true population mean,

but it's not going to be that helpful.

So there's a balancing act going on.

You want a confidence interval that's narrow enough to be useful, but wide enough that

the true population mean will usually be inside a confidence interval of that percent.

We can't always have large samples.

It's often the case that there's not enough time or money to collect 100s of data points

to calculate a confidence interval.

With small sample sizes, the distribution of sample means isn't always exactly normal,

so we often use a t-distribution instead of a z-distribution to find out where the middle

95% of our data is.

The t-distribution, like the z-distribution, is a continuous probability distribution that's unimodal.

It's a useful way to represent sampling distributions.

The t-distribution changes its shape according to how much information there is.

With small sample sizes there's less information so the t-distribution has thicker tails to

represent that our estimates are more uncertain when there's not much data.

However as we get more and more data, the t-distribution becomes identical to the z-distribution.

Generally, sample sizes that are greater than 30 are considered “large enough” because

scientists generally believe that sampling distributions where the sample is 30+ are

close enough to normal...though 30 is an arbitrary cutoff just like 0.05.

However, when we're estimating population proportions, like the proportion of people

who are color blind, the general rule is that your sample size need to be big enough so

that on average, you'd expect to get at least 10 colorblind, and at least 10 non-colorblind people.

For similar reasons, most people consider that “close enough”.

Since about 8% of males are colorblind, if I only had a sample of 50 males, on average

I'd expect around 4 males per group to be color blind, so my sample size wouldn't

be quite big enough to assume it's normal.

Instead I'd use the almost normal t-distribution.

If a drug that's being developed claimed to reduce the proportion of colorblind males

born to mothers who took it, we could take a sample of 50 male infants to see if the

proportion of colorblindness is different from 8%.

Though colorblindness isn't usually life threatening, it can be inconvenient, so you

decide to calculate a confidence interval to see if it's likely to be effective.

After randomly selecting 50 male infants from mothers who took the drug, you calculate the

sample proportion of colorblind infants, which is 6%, and calculate the distribution of sample

proportions which has a mean of 6%--the same as the sample mean--and a standard error of 0.033.

Since our sample size isn't big enough to assume that the distribution of sample proportions

is shaped like the z-distribution, we can use the t-distribution to calculate the range

of our 95% confidence interval.

I mentioned before that the t-distribution's shape changes with how much data we have.

We'll talk more in detail later as to how to choose the right t-distribution, but for

now, we'll use this one:

While t-score tables do exist, it's often easier to have a statistical program calculate

the t-values that correspond to the 2.5th and 97.5th percentiles, since there are many

different t-distributions.

Your computer tells you that the t-values corresponding to those percentiles are 2.01 and -2.01.

And to convert to a raw score from a t-score, we again use this formula, just with a t-score

instead of a z-score.

Our confidence interval for proportion of colorblind males is -0.6% to 12.63%.

8% is inside our confidence interval, so it's not too much of a stretch to think that 8%

could be the true population proportion, even though we only observed a sample proportion of 6%.

Based on this confidence interval we don't have any evidence to conclude whether this

medicine is effective or not.

So since the company researching the drug is pretty cautious, they decide not to go

ahead with it.

One place you may have seen confidence intervals “in the wild” is in the news during election season.

When newscasters report results from exit polls they'll usually say something like

“Candidate A is tracking at 64%, with a margin of error of 3 %” or you may see a

chart like this:

The margin or error is usually telling you how far the bounds of the confidence interval

are from the mean, and is represented by this part of the confidence interval formula:

The margin of error, just like a confidence interval, reflects the uncertainty that surrounds

sample estimates of parameters like the mean or a proportion.

If a poll shows that a Presidential candidate is tracking at of 64% of the vote, plus or

minus 3%, we shouldn't be surprised if it turns out that the true vote was 61%, since

that's within the margin or error.

You can think of values inside the margin of error or confidence interval as values

that might be reasonable estimates of the true population parameter.

Confidence intervals quantify our uncertainty.

They also demonstrate the tradeoff of accuracy for precision.

A 100% confidence interval will always contain the true population mean, but it's useless.

We have to sacrifice a little bit of accuracy in order to gain more precision.

A 99% confidence interval will give us a more useful range since it won't be infinitely

long..., but It's now possible that our confidence interval won't contain the true mean.

And you've probably encountered this tradeoff in your daily life.

Say you're running a marathon (like everybody does) and you want to load up your iPhone

with music, but you don't know how long you're going to take, you could buy 150

songs on iTunes, which is expensive, or you could buy only 70 and have a chance of running

out of music.

You increase your risk of not having enough, but then again you're saving yourself from

having to buy 80 extra songs…

Maybe it's time for a streaming service?

Confidence intervals demonstrate this delicate balancing act... and help us understand how

to hit the sweet spot of information vs. accuracy.

Thanks for watching, I'll see you next time in my gold lame pants.