Lies. Damn lies. And statistics

Stats gets a bad rap.

And sometimes it makes sense why.

We've talked a lot about how p-values let us know something significant in our data--but

those p-values and the data behind them can be manipulated.

Hacked.

P hacked.

P-hacking is manipulating data or analyses to artificially get significant p-values.

Today we're going to take a break from learning new statistical models, and instead look at

some statistics gone wrong.

And maybe also some props gone wrong.

INTRO

To recap to calculate a p-value, we look at the Null Hypothesis--which is the idea that

there's no effect.

This can be no effect of shoe color on the number of steps you walked today, or no effect

of grams of fat in your diet on energy levels.

Whatever it is, we set this hypothesis up just so that we can try to shoot it down.

In the NHST framework we either reject, or fail to reject the null.

This binary decision process leads us to 4 possible scenarios:

The null is true and we correctly fail to reject it

The null is true but we incorrectly reject it.

The null is false and we correctly reject it.

The null is false and we incorrectly fail to reject it.

Out of these four options, scientists who expect to see a relationship are usually hoping

for this one.

In NHST, failing to reject the null is a lack of any evidence, not evidence that nothing

happened.

So scientists and researchers are incentivised to find something significant.

Academic journals don't want to publish a result saying: “We don't have convincing

evidence that chocolate cures cancer but also we don't have convincing evidence that it doesn't".

Popular websites don't want that either.

That's like anti-clickbait.

In science, being able to publish your results is your ticket to job stability, a higher

salary, and prestige.

In this quest to achieve positive results, sometimes things can go wrong.

P-hacking is when analyses are being chosen based on what makes the p-value significant,

not what's the best analysis plan.

Statistical tests that look normal on the surface may have been p-hacked.

And we should be careful when consuming or doing research so that we're not misled

by p-hacked analyses.

“P-hacking” isn't always malicious.

It could come from a gap in a researcher's statistical knowledge, a well-intentioned

belief in a specific scientific theory, or just an honest mistake.

Regardless of what's behind p-hacking, it's a problem.

Much of scientific theory is based on p-values.

Ideally, we should choose which analyses we're going to do before we see the data.

And even then, we accept that sometimes we'll get a significant result even if there's

no real effect, just by chance.

It's a risk we take when we use Null Hypothesis Significance Testing.

But we don't want researchers to intentionally create effects that look significant, even

when they're not.

When scientists p-hack, they're often putting out research results that just aren't real.

And the ramifications of these incorrect studies can be small--like convincing people that

eating chocolate will cause weight loss--to very, very serious--like contributing to a

study that convinced many people to stop vaccinating their kids.

Analyses can be complicated.

For example, x-k-c-d had a comic associating jelly beans and acne.

So you grab a box of jelly beans and get experimenting.

It turns out that you get a p-value that's greater than 0.05.

Since your alpha cutoff is 0.05, you fail to reject the null that jelly beans are not

associated with breakouts.

But the comic goes on there are different COLORS of jelly beans.

Maybe it's only one color that's linked with acne!

So you go off to the lab to test the twenty different colors.

And the green ones produce a significant p-value!

But before you run off to the newspapers to tell everyone to stop eating green jelly beans,

let's think about what happened.

We know that there's a 5% chance of getting a p-value less than 0.05, even if no color

of jelly bean is actually linked to acne.

That's a 1 in 20 chance.

And we just did 20 separate tests.

So what's the likelihood here that we'd incorrectly reject the null?

Turns out with 20 tests--it's way higher than 5%.

If jelly beans are not linked with acne, then each individual test has a 5% chance of being

significant, and a 95% chance of not being significant.

So the probability of having NONE of our 20 tests come up significant is 0.95 to the twentieth

power, or about 36%.

That means that about 64% of the time, 1 or more of these test will be significant, just

by chance, even though jelly beans have no effect on acne.

And 64% is a lot higher than the 5% chance you may have been expecting.

This inflated Type I error rate is called the Family Wise Error rate.

When doing multiple related tests, or even multiple follow up comparisons on a significant

ANOVA test, Family Wise Error rates can go up quite a lot.

Which means that if the null is true, we're going to get a LOT more significant results

than our prescribed Type I error rate of 5% implies.

If you're a researcher who put a lot of heart, time, and effort into doing a study

similar to our jelly bean one, and you found a non-significant overall effect, that's

pretty rough. Dissapointing.

No one is likely to publish your non-results.

But we don't want to just keep running tests until we find something significant.

A Cornell food science lab was studying the effects of the price of a buffet on the amount

people ate at that buffet.

They set up a buffet and charged half the people full price, and gave the other half

a 50% discount.

The experiment tracked what people ate, how much they ate, and who they ate it with, and

had them fill out a long questionnaire.

The original hypothesis was that there is an effect of buffet price on the amount that

people eat.

But after running their planned analysis, it turned out that there wasn't a statistically

significant difference.

So, according to emails published by Buzzfeed, the head of the lab encouraged another lab

member to do some digging and look at all sorts of smaller groups.

“males, females, lunch goers, dinner goers, people sitting alone, people eating in groups

of 2, people eating in groups of 2+, people who order alcohol, people who order soft drinks,

people who sit close to buffet, people who sit far away…”

According to those same emails, they also tested these groups on several different variables

like “[number of] pieces of pizza, [number of] trips, fill level of plate, did they get

dessert, did they order a drink...”

Results from this study were eventually published in 4 different papers.

And got media attention.

But one was later retracted and 3 of the papers had corrections issued because of accusations

of p-hacking and other unethical data practices.

The fact that there were a few, out of many, statistical tests conducted by this team that

were statistically significant is no surprise.

Many researchers have criticized these results.

Just like in our fake jelly bean experiment, they created a huge number of possible tests.

And even if buffet price had no effect on the eating habits of buffet goers, we know

that some, if not many, of these tests were likely to be significant just by chance.

And the more analyses that were conducted, the more likely finding those fluke results becomes.

By the time you do 14 separate tests, it's more likely than not that you'll get at

LEAST one statistically significant result, even if there's nothing there.

The main problem arises when those few significant results are reported without the context of

all the non-significant ones.

Let's pretend that you make firecrackers.

And you're new to making fire crackers. You're not great at it. And sometimes make mistakes that cause

the crackers to fizzle when they should go “BOOM”.

You make one batch of 100 firecrackers and only 5 of them work.

You take those 5 exploded firecrackers (with video proof that they really went off) to

a business meeting to try to convince some Venture Capitalists to give you some money

to grow your business.

Conveniently, they don't ask whether you made any other failed firecrackers.

They think you're showing them everything you made.

And you start to feel a little bad about taking their million dollars.

Instead, you do the right thing, and tell them that you actually made 100 firecrackers

and these are just the ones that turned out okay.

Once they know that 95 of the firecrackers that you made failed, they're not

going to give you money.

Multiple statistical tests on the same data are similar.

Significant results usually indicate to us that something interesting could be happening.

That's why we use significance tests.

But if you see only 5 out of 100 tests are significant you're probably going to be

a bit more suspicious that those significant results are false positives.

Those 5 good firecrackers may have just been good luck.

When researchers conduct many statistical tests, but only report the significant ones,

it's misleading.

Depending on how transparent they are, it can even seem like they only ran 5 tests,

of which 5 were significant.

There is a way to account for Family Wise Errors.

The world is complex, and sometimes so are the experiments that we use to explore it.

While it's important for people doing research to define the hypotheses they're going to

test before they look at any data, it's understandable that during the course of the

experiment they may get new ideas.

One simple way around this is to correct for the inflation in your Family Wise Error rate.

If you want the overall Type I error rate for all your tests to be 5%, then you can

adjust your p-values accordingly.

One very simple way to do this is to apply a Bonferroni correction.

Instead of setting a usual threshold--like 0.05--to decide when a p-value is significant

or non-significant, you take the usual threshold and divide it by the number of tests you're doing.

If we wanted to test the effect of 5 different health measures on risk of stroke, we would

take our original threshold--0.05--and divide by 5.

That leaves us with a new cutoff of 0.01.

So in order to determine if the effect of hours of exercise--or any of our other 4 measures--has

a significant effect on your risk of stroke, you would need to have a p-value of below 0.01

instead of 0.05.

This may seem like a lot of hoopla over a few extra statistical tests, but making sure

that we limit the likelihood of putting out false research is really important.

We always want to put out good research, and as much as possible, we want the results we

publish to be correct.

If you don't do research yourself, these problems can seem far removed from your everyday

life, but they still affect you.

These results might affect the amount of chemicals that are allowed in your food and water, or

laws that politicians are writing.

And spotting questionable science means you not have to avoid those green jelly beans.

Cause green jelly beans are clearly the best.

Thanks for watching, I'll see you next time.