Statistics 101: What Is Hypothesis Testing and Statistical Significance

William Spaniel Author

05/31/2016 Added

993 Plays

Description

This video is designed to help understand what hypothesis testing and statistical significance is in quantitative research.

Searchable Transcript

Search:

[00:00:00.963]Hi, I'm William Spaniel.
[00:00:02.652]Let's talk learn about some statistics.
[00:00:04.300]Today, I want to go over the very basics of
[00:00:06.053]proportion testing and talk about what the intuition behind
[00:00:08.700]what statistical significance is.
[00:00:11.242]So there's actually many uses for proportion testing.
[00:00:14.040]You can look to see if your favorite sports team
[00:00:16.514]is really better than everybody else.
[00:00:18.237]You can check to see whether the coin
[00:00:20.357]that you're flipping is actually fair
[00:00:21.781]or weighted equally so it comes up head and tails
[00:00:24.034]equal amounts of times.
[00:00:25.963]You can see if a bill is likely to pass a vote
[00:00:29.270]in a popular election or something like that,
[00:00:31.811]and you can also talk about the mortality rate
[00:00:34.249]of a particular disease so it's very versatile.
[00:00:36.931]You'll see this coming up in a lot of
[00:00:38.208]different academic writings and studies, and so forth,
[00:00:40.797]and so, it's important to actually understand what they mean
[00:00:42.992]when they talk about significance in this context.
[00:00:46.614]So we'll actually explain how to conduct these test
[00:00:49.801]in later videos but right now, I just want to go over,
[00:00:52.060]as I said, with the intuition behind what this hypothesis
[00:00:55.009]testing and statistical significance really means.
[00:00:57.813]And to do this, I'm going to use baseball as an example.
[00:01:00.171]There's a long running theory that the home team
[00:01:01.726]has an advantage in baseball but you know,
[00:01:03.572]how can we really be sure?
[00:01:05.593]And well, one way we could track is by gathering data
[00:01:08.390]so in the 2009 season, the home team won 1,333 games
[00:01:13.099]out of 2430 total and what we want
[00:01:17.215]to find out here is if this information that we have
[00:01:19.955]is indicative of a pattern or if it's just a coincidence.
[00:01:23.183]Our default assumption or what we call
[00:01:24.607]a null hypothesis right here
[00:01:26.499]is that the home team wins 50% of the game.
[00:01:29.065]That's essentially saying that there's no advantage
[00:01:31.085]to being the home team and no disadvantage
[00:01:32.780]to being the home team either.
[00:01:34.799]That means it (mumbles) the data to approve
[00:01:36.651]that something else is going on here,
[00:01:38.276]so it's sort of like in a trial, you have to have
[00:01:42.711]beyond a reasonable doubt.
[00:01:43.895]Essentially, what we're creating here is this definition
[00:01:46.530]of what reasonable doubt is applying it rigorously
[00:01:50.014]using some statistic.
[00:01:51.139]So if we can't reject this null hypothesis,
[00:01:53.659]if we can't prove that this is not true
[00:01:55.331]beyond a reasonable doubt,
[00:01:56.043]well, we kind of just ignore it
[00:01:57.629]and to chalk it up those coincidence.
[00:02:00.501]So that's that, now let's get to some issues here.
[00:02:04.598]One problem with just looking at numbers and saying,
[00:02:07.188]well, yeah, this is indicative of a pattern is that
[00:02:09.289]data itself can be very tricky.
[00:02:10.775]Suppose that we only looked at three games,
[00:02:12.551]and the home team won all of them?
[00:02:14.072]You might wonder, well, is that indicative of something?
[00:02:16.476]But, well you know, not really, if no team
[00:02:19.064]were to have an advantage whether you're on the road
[00:02:21.619]or at home, so if the home team had no advantage,
[00:02:23.895]their expectation was they're going to win
[00:02:25.420]half the games at home anyway.
[00:02:26.929]It's easily possible for the home team to win
[00:02:28.868]all three of those games.
[00:02:30.946]But then, we get into a problem of what exactly qualifies
[00:02:33.582]as solid evidence?
[00:02:34.604]What if they went 17 and 10?
[00:02:35.899]Obviously, that's a worse winning percentage
[00:02:38.012]but here we have a larger sample,
[00:02:39.939]so is that indicative of something?
[00:02:41.751]Well, what about if they went to 170 and 100?
[00:02:45.500]Now, you have a much larger sample.
[00:02:47.218]It's the same percentage as the time where they only went
[00:02:49.297]17 and 10 but you know, we need
[00:02:51.782]a rigorous way of tracking this process.
[00:02:54.336]Unfortunately, there's an important finding in statistics
[00:02:56.344]that allows us to create some solid rules
[00:02:58.620]presiding whether or not something is significant.
[00:03:01.005]And there's three things that go into determining
[00:03:03.501]whether a finding is statistically significant.
[00:03:05.881]The first is the observed percentage.
[00:03:12.700]So in our example, we were looking at baseball teams
[00:03:15.166]and we saw that the home team went about 55% of the games.
[00:03:19.054]This isn't very far away from 50%.
[00:03:21.256]It's only five percent different.
[00:03:22.517]So you know, maybe our results might not be conclusive
[00:03:25.346]because it's so close to that null hypothesis of 50%.
[00:03:31.725]The second important element is the number of observations
[00:03:33.570]that you have in your study.
[00:03:35.091]It's also called the sample size
[00:03:36.252]or the size of your population of your sample.
[00:03:39.339]So the more data points you have, the more likely
[00:03:41.781]your result is to be statistically significant
[00:03:43.936]and the null hypothesis is actually wrong.
[00:03:46.269]So you can think of this as it's regressing toward the mean
[00:03:49.617]or something like that.
[00:03:50.626]You can go gambling in Vegas and you can win a lot of money
[00:03:53.704]in the short term but you're not going to be able to
[00:03:55.724]sustain your luck forever.
[00:03:57.082]Eventually, reality is going to bring you down
[00:03:58.742]to whatever your average win percentage is.
[00:04:00.937]And so, that's really what's going on here with
[00:04:03.201]the more data you have, the more confident you can be
[00:04:05.580]that whatever percentage that you've observed is actually
[00:04:07.798]closer to the actual percentage,
[00:04:09.644]so you're reducing your sort of margin of error so to speak,
[00:04:14.404]and that's what's going to be helping you out
[00:04:16.726]in the second part here.
[00:04:18.201]So in the baseball example, you observed 2430 games
[00:04:21.858]and that's a lot so despite the fact that our
[00:04:24.400]observed proportion was so close to the null hypothesis,
[00:04:27.710]the 54.9% observed percentage was very close
[00:04:31.731]to our null hypothesis of 50%.
[00:04:33.774]You know, maybe our results will be
[00:04:35.179]statistically significant after all.
[00:04:37.640]And finally, the third element has to deal with
[00:04:39.672]what the null hypothesis was.
[00:04:41.320]The closer it is to 50% either coming from above or below,
[00:04:45.298]the less likely the result
[00:04:46.536]is to be statistically significant.
[00:04:48.521]This is a little bit technical, unlike the other two
[00:04:50.578]which had a lot more of a straightforward reason
[00:04:52.981]why that's the case.
[00:04:53.944]But here the reason is that one of the ways
[00:04:56.324]that you calculate statistical significance
[00:04:59.099]is using the standard deviation,
[00:05:00.841]and the larger the standard deviation is the harder it is
[00:05:04.226]for something to become statistically significant.
[00:05:06.246]So the standard deviation as it turns out
[00:05:08.251]when you're dealing with percentages is largest at 50%
[00:05:11.629]and smallest at zero and a 100%,
[00:05:13.753]and as you move away from 50% in either direction,
[00:05:16.193]it becomes smaller which decreases your margin of error
[00:05:19.838]what we call a confidence interval to be technical
[00:05:22.903]and that will make it harder for you to show
[00:05:25.326]that something is statistically significant.
[00:05:27.103]So in our baseball example, the null hypothesis was,
[00:05:30.017]you know, there's no advantage for the home team
[00:05:31.406]and no disadvantage, so your null hypothesis here is 50%
[00:05:34.982]and we have no help here unfortunately as a result.
[00:05:39.103]So I actually ran this test in my own time
[00:05:41.147]and not presenting the math behind it quite yet.
[00:05:42.976]We'll look at that later in a different video
[00:05:46.053]but I found out that this was actually better
[00:05:48.200]than 99% statistically significant so,
[00:05:51.021]what is the intuition behind that statement?
[00:05:52.798]What does that mean when I say that
[00:05:55.146]this test that I ran came out as more than 99% significant.
[00:05:58.552]Well, it means that if we were to rerun the 2009 season,
[00:06:01.908]so if essentially, we could go back in time
[00:06:04.044]and play that season over again, you know,
[00:06:06.308]and we get different results because you know,
[00:06:08.050]baseball is a bunch of chance and so,
[00:06:10.407]whatever happen in one game
[00:06:11.788]might not happen again in the next game.
[00:06:13.915]What that means by saying that this is
[00:06:15.486]99% statistically significant is that
[00:06:17.738]if we redid the 2009 season, just you know a bunch of times,
[00:06:22.378]an arbitrarily large number of times,
[00:06:24.271]then the expectation is that 99% of those seasons
[00:06:27.696]would come out at least 99% because I said,
[00:06:30.134]this is more than 99% of statistically significant.
[00:06:34.209]So at least 99% of those games, oh, sorry, those seasons
[00:06:38.238]that are played would have results where the home team
[00:06:40.880]wins more than half of the games.
[00:06:43.602]So basically what we're saying here is that this result
[00:06:45.703]is really robust and if we went back and redid that data
[00:06:48.862]again, and again, and again, and again, and again,
[00:06:51.021]we'd see similar results where we keep getting results
[00:06:53.564]that are getting results that are greater than 50%.
[00:06:56.257]Now, there are three types of statistical significance
[00:06:58.080]that you'll frequently see in the literature
[00:07:00.680]and in academic journals and so forth,
[00:07:02.480]you'll see something called 90% statistical significance,
[00:07:04.640]95% statistical significance,
[00:07:06.462]and 99% statistical significance.
[00:07:08.680]Now, all of those percentages are simply analogies
[00:07:10.665]to what I said earlier so if something came up
[00:07:12.604]90% statistically significant, that means
[00:07:14.763]if you reran the data generating process a bunch of times,
[00:07:17.701]then 90% of those times would come up greater than
[00:07:20.499]or less than, depending upon what you're looking at
[00:07:23.018]your null hypothesis.
[00:07:25.340]So 90% really isn't that good so when we see something
[00:07:28.080]with 90% statistical significance,
[00:07:30.268]the best recommendation I can give is to only leave
[00:07:33.635]those results if you have a good theory behind them.
[00:07:35.690]So in this baseball example,
[00:07:37.152]if we had a 90% statistical significance,
[00:07:39.649]you know, we might be willing to accept the fact
[00:07:41.471]that the home team does have an advantage because
[00:07:45.593]we have a good theory why that would be the case.
[00:07:47.511]They bat last which is a strategic advantage,
[00:07:49.856]they know how many runs they need,
[00:07:51.566]whereas the away team doesn't bat last
[00:07:53.505]so they don't know that.
[00:07:54.736]And the home team also doesn't have to travel
[00:07:56.245]so they're going to be less tired.
[00:07:57.255]So there's a good theoretical reason to believe that
[00:07:59.190]the home team in baseball has an advantage
[00:08:01.687]over the away team but not all things
[00:08:04.044]that you'll be looking at will have that
[00:08:06.285]sort of theory behind it.
[00:08:08.026]We might not know what's the case.
[00:08:09.489]We might not have any intuition or any good guess
[00:08:12.066]beforehand, before we conduct these studies,
[00:08:14.852]so you know, when you have a 90% statistical significance,
[00:08:17.848]you really have to be cautious about your results there
[00:08:20.170]because it's very easy to come up with a false positive.
[00:08:22.608]95% statistical significance, here,
[00:08:26.625]if you don't already have a good theory behind this result,
[00:08:29.272]you should start considering so it's unlikely
[00:08:32.012]that you'll be getting false positives here,
[00:08:33.591]it's still possible of course but,
[00:08:35.611]it's very unlikely.
[00:08:37.004]And so, you should really start considering some theories
[00:08:39.745]behind your results if you don't already have one.
[00:08:42.125]And finally, you'll also see 99% statistical significance
[00:08:45.329]and here, it's just extremely unlikely that the results
[00:08:48.127]that we're getting are purely coincidental.
[00:08:49.625]So at that point, you definitely
[00:08:51.006]have to start considering theories.
[00:08:52.458]Again, it's possible because
[00:08:54.456]it's not 100% statistically significant that you'll get
[00:08:58.670]a false positive here with this 99% statistical significance
[00:09:01.654]but it's just very unlikely that your results
[00:09:03.674]are going to be giving you, sorry,
[00:09:06.077]your data will be giving you those resuts
[00:09:07.354]if your null hypothesis were true.
[00:09:10.454]So you really need to start considering other theories
[00:09:13.612]or theories behind what your results are
[00:09:15.098]if you get something that's 99% statistically significant.
[00:09:18.256]All right, that wraps that up.
[00:09:19.870]So we just had a quick look at what the meaning is
[00:09:22.319]behind hypothesis testing and statistical significance,
[00:09:25.419]and what we're looking at proportions or percentages
[00:09:27.649]like we were with this baseball example in win percentages.
[00:09:30.772]Join me later when we start looking at
[00:09:32.095]some actual calculations on how to do the mathematics
[00:09:34.138]behind these formulations and the calculation
[00:09:37.737]behind statistical significance in proportion testing.

The screen size you are trying to search captions on is too small!

You can always jump over to MediaHub and check it out there.

Comments

0 Comments

Statistics 101: What Is Hypothesis Testing and Statistical Significance

Description

Searchable Transcript

Comments icon comment

Related Channels

Comments