Hypothesis Testing

Module 1

What is Hypothesis Testing?

Core idea: Hypothesis testing is a formal method for using sample data to decide whether there is enough evidence to reject a particular claim about a population. We never "prove" anything — we are finding how unlikely (low probability) the actual outcome of our test has been if we assume the null hypothesis was true. We are forced to take a sample and make a test because it is usually too expensive, too time-consuming, or not feasible (in destructive testing) to test every item in the population. We are taking a sample and trying to make some judgement about the whole population from the sample measurement.

A statistic is any numerical quantity calculated solely from the values in a sample. Examples include the sample mean, sample standard deviation, sample proportion and sample correlation coefficient.

This contrasts with a parameter, which is a numerical quantity describing an entire population. For example, if you measure the heights of 50 students chosen from a whole school:

The average height of those 50 students is a sample statistic.
The average height of all students in the school is a population parameter.

Two starting situations

A starting point for hypothesis testing is usually one of two situations: we want to see i) whether some change we have made to a process or situation has produced the effect we intended or ii) whether some claim about a population — e.g. the mean value of some property such as mass or length — is actually true.

Situation 1

Some process has been operating for a long time and its behaviour is well understood — for example, the proportion of faulty items or the mean output is known to be stable. A change is made to try to improve the process, data are collected from the modified process, and we calculate statistics from the sample to test whether the change has had the desired effect.

Situation 2

A claim is made about a population parameter — for example, that the mean mass of products is 350 g or that for an item selected at random from a population the probability the selected item is faulty is 0.02 — and we wish to test whether the available evidence supports that claim.

In both situations we take a sample and calculate one or more statistics from the sample data.

Before moving to formal procedures, it helps to understand why hypothesis testing exists — what real problem it solves, and how it connects populations to samples.

Populations and samples

Suppose we want to know something about a whole population — for example, the mean weight of all adults in a city, or the proportion of defective items from a production line. Measuring every member of the population is usually too slow, too expensive, or genuinely impossible. So instead we take a sample and use it to draw conclusions about the population.

The sample statistic will almost certainly differ from the true population parameter — even when nothing has changed — simply due to the randomness of sampling. The key question is therefore:

\[\text{Is the difference between the sample statistic and the population parameter due to chance, or is it real?}\]

Three motivating scenarios

Scenario 1 — A suspicious coin

A fair coin should satisfy \(P(\text{Head}) = P(\text{Tail}) = \tfrac{1}{2}\). If we flip it five times and get five heads, the probability of that happening with a fair coin is: \[\left(\tfrac{1}{2}\right)^5 = \tfrac{1}{32} \approx 3.1\%\] That's unusual but possible. If we got 10 heads in a row the probability drops to about 0.1%. At some point the result becomes so unlikely under the fair-coin assumption that we lose confidence in that assumption and prefer an alternative explanation: the coin is biased towards heads.

This is the logic of hypothesis testing. We start with the null hypothesis \(H_0: p = 0.5\) (fair coin). The significance level — say 5% — is the threshold below which we judge the evidence "too unlikely to be chance". If the observed result has probability \(\leq 5\%\) under \(H_0\), we reject \(H_0\) and accept \(H_1: p > 0.5\) (biased towards heads).

Scenario 2 — Disks in a bag

Suppose we are told there are equal numbers of red, blue and green disks in a bag, so \(P(\text{Red}) = \tfrac{1}{3}\). We cannot count every disk (the total is the population) but we can draw a sample.

If we draw 30 disks and find 18 are red, is that consistent with \(P(\text{Red}) = \tfrac{1}{3}\)? We'd expect about 10 red disks if the claim were true. Having 18 might simply be bad luck — or it might indicate there are actually more red disks than claimed. The hypothesis test tells us whether the sample provides sufficient evidence to doubt the original claim.

Notice the key logical move: we calculate the probability of the observed result (or worse) assuming \(H_0\) is true. We can't test \(H_1\) directly because it doesn't specify an exact value of \(p\).

Hypothesis testing arises naturally whenever it is impossible or impractical to measure the whole population:

Situation	Population	Why we can't measure it all
Destructive testing — steel beams tested to breaking point	All beams	Testing destroys them; nothing left to use
Quality control — proportion of defective items from a machine	All items ever produced	Production is ongoing; too slow and costly
Medical trials — does a new drug improve cure rates?	All future patients	Cannot wait to treat everyone before deciding
Environmental testing — do new engines reduce emissions?	All engines ever made	Too many; sampling is the only feasible approach

In every case the same question arises: is the difference we see in the sample real, or just random variation? Hypothesis testing provides a rigorous, agreed framework for answering it.

Think about it

Why is the difference between sample and population statistics unavoidable?

Different random samples from the same population will give different statistics — this is called sampling variability. Even a completely honest sample will rarely match the population parameter exactly, purely by chance. This means any observed difference could in principle be nothing more than this natural variability. The hypothesis test exists to judge how much "weight" that explanation carries.

What if we took the entire population as our sample?

If you could measure every member of the population there would be no uncertainty: the sample statistic would equal the population parameter exactly and you would not need a hypothesis test at all. It is precisely because we can't do this (for the reasons in the table above) that statistical inference — and hypothesis testing — become necessary.

Module 2

First Attempts

Before learning the formal vocabulary, let's see the core reasoning of a hypothesis test in action.

The Basic Framework

State the hypotheses. Set up a null hypothesis \(H_0\) (the claim to test) and an alternative hypothesis \(H_1\) (what we believe if \(H_0\) is wrong).

Choose a significance level. Decide, before collecting data, how small a probability counts as "surprising" — typically \(\alpha = 0.05\) or \(\alpha = 0.01\).

Identify the test statistic and its distribution under \(H_0\). For A-level, this is usually a count \(X \sim B(n,p)\).

Calculate the probability of observing a result as extreme as (or more extreme than) the observed value, assuming \(H_0\).

Compare with \(\alpha\). If this probability \(\leq \alpha\), reject \(H_0\). Otherwise, do not reject \(H_0\).

State a conclusion in context. Always refer back to the original situation.

Example outcomes:
In Situation 1 we may conclude that a change made to a manufacturing process has significantly reduced the proportion of faulty products.
In Situation 2 we may conclude that there is sufficient evidence to reject a claim such as the mean mass of products being 350 g.

Important: Rejecting the null hypothesis does not prove that it is false. The observed sample and statistic could still have arisen purely by chance even when the null hypothesis is true. However, if such an outcome would be sufficiently unlikely — less likely than the chosen significance level — we regard the evidence as strong enough to reject \(H_0\).

In other words, there is always some possibility that we reject a null hypothesis that is actually true. The significance level controls how willing we are to accept that risk.

Suppose a manufacturer claims that exactly 30% of the sweets in a bag are red. You open a bag of 20 sweets and find only 2 are red. Is this just bad luck, or is the 30% claim actually wrong? Hypothesis testing gives us a rigorous way to answer this question — we'll return to this exact example once we reach the One-Tailed Tests module.

The key idea is: assume the claim is true, then ask — how probable is it to observe something as extreme as (or more extreme than) what we actually saw? If that probability is very small, we have grounds to doubt the claim.

Why "do not reject" rather than "accept"? We never "prove" the null hypothesis — we only judge whether the evidence is strong enough to doubt it. (More on this in the Language module.)

Think about it

Why do we assume \(H_0\) is true to begin with?

The logic mirrors a legal presumption of innocence. We assume the null (the "status quo") and ask: if it really were true, how likely is the observed data? Only if the data is highly unlikely under \(H_0\) do we have grounds to reject it. This structure means the burden of proof lies with the data, not with the hypothesis.

What does "statistically significant" mean?

"Statistically significant" simply means the observed result is unlikely enough (under \(H_0\)) to fall below the chosen significance level \(\alpha\). It does not automatically mean the result is practically important or large — a tiny real-world difference can be statistically significant with a large enough sample.

Module 3

The Language of Hypothesis Testing

This module explains each term in context first; a summary table of all the vocabulary appears at the end of the page for quick reference.

The null hypothesis and the p-value

The null hypothesis, denoted \(H_0\), represents the original situation or claim that we assume to be true unless the sample provides sufficiently strong evidence against it.

In Situation 1, the null hypothesis is usually that the process has not changed and that any differences observed in the sample are due solely to random variation.
In Situation 2, the null hypothesis is that the claim we have been given is true.

We then determine how unusual the observed statistic would be if the null hypothesis were true. This is done by calculating a p-value: the probability of obtaining the observed value of the statistic, or a value even more extreme, assuming the null hypothesis is true.

Even if the null hypothesis is true, random sampling variation means that different samples will produce different statistics. The observed statistic may therefore differ from what we expected purely by chance. The question is whether the difference is small enough to be reasonably explained by chance variation, or so large that it provides evidence against the null hypothesis.

If the p-value is less than the chosen significance level, the observed result is considered sufficiently inconsistent with the null hypothesis that we reject it.

Setting up the hypotheses

For binomial tests, the parameter being tested is the probability of success \(p\). We write \(p_0\) for the specific numerical value stated in the null hypothesis — for example, if \(H_0\) claims the probability is 0.3, then \(p_0 = 0.3\). The null hypothesis always takes the form:

\[H_0 : p = p_0\]

The alternative hypothesis is chosen based on the research question, before looking at the data:

One-tailed (lower):
\(H_1 : p < p_0\)

Use when there is prior reason to suspect the true probability is smaller than \(p_0\).

One-tailed (upper):
\(H_1 : p > p_0\)

Use when there is prior reason to suspect the true probability is larger than \(p_0\).

Two-tailed: \(H_1 : p \neq p_0\) — use when the question asks whether the probability has changed from \(p_0\), with no prior reason to expect a particular direction.

Common errors in writing hypotheses:
✗ Using the sample statistic: e.g. writing \(H_0: \bar{x} = 50\). The null hypothesis is always a statement about the population parameter, not the sample. The correct form is \(H_0: \mu = 50\), where \(\mu\) is the population mean.

✗ Using an inequality in \(H_0\): e.g. writing \(H_0: p \leq 0.3\). The null hypothesis must state an exact value so that the probability distribution of the test statistic is fully determined. Always use \(H_0: p = 0.3\).

The Binomial Model

When testing a proportion, the test statistic \(X\) counts the number of successes in \(n\) independent trials, each with probability \(p\) of success. Under \(H_0\), the exact value \(p_0\) is assumed to be true, so:

\[X \sim B(n,\, p_0)\]

We observe a value \(x\) and compute the p-value — the probability of getting a result at least as extreme as \(x\) in the direction of \(H_1\). This p-value is then compared with \(\alpha\) to reach a decision. The full procedure is covered in the next modules.

Now that you've seen these ideas in context, here's the full vocabulary in one place. Getting the language right is essential — especially in exam answers, where every word in the conclusion carries marks.

Term	Meaning
Null hypothesis \(H_0\)	The default assumption about the population parameter. Always states an exact value, e.g. \(p = 0.3\).
Alternative hypothesis \(H_1\)	What we conclude if \(H_0\) is rejected. States the direction of change (or simply that a change has occurred). Determines whether the test is one- or two-tailed.
Test statistic	A numerical value calculated from the sample data that is used to decide between \(H_0\) and \(H_1\). Its probability distribution under \(H_0\) must be known. Examples: the count of successes \(X\) in a binomial test; the sample mean \(\bar{X}\) in a Normal mean test; the sample PMCC \(r\) in a correlation test.
p-value	The probability, calculated assuming \(H_0\) is true, of obtaining a test statistic at least as extreme as the one observed. A small p-value means the observed result would be unlikely if \(H_0\) were true.
Significance level \(\alpha\)	The threshold probability chosen before the test. If the p-value falls below \(\alpha\), the result is declared significant and \(H_0\) is rejected. Common choices are 5% and 1%.
Critical region	The set of values of the test statistic for which \(H_0\) is rejected — i.e. the values so extreme that they fall below the significance level.
Critical value	The boundary of the critical region. The test statistic must be at least as extreme as the critical value in order to reject \(H_0\).
Reject \(H_0\)	The p-value is less than \(\alpha\): the observed result is sufficiently unlikely under \(H_0\) that we conclude \(H_1\) is supported. We say the result is statistically significant at level \(\alpha\).
Do not reject \(H_0\)	The p-value is greater than or equal to \(\alpha\): insufficient evidence to favour \(H_1\). This does not mean \(H_0\) is proved true — only that the data are consistent with it.
Type I error	Rejecting \(H_0\) when it is actually true — a "false alarm". The probability of a Type I error is exactly \(\alpha\), the significance level.
Type II error	Failing to reject \(H_0\) when it is actually false — a "missed detection". Its probability depends on the true value of the parameter and is not fixed by \(\alpha\) alone.

Think about it

Why must \(H_0\) contain an equals sign?

To calculate the p-value we need to know the probability distribution of the test statistic exactly. An exact value — e.g. \(p = 0.3\) — fully determines the distribution \(B(n, 0.3)\), so every probability can be computed. An inequality like \(p \leq 0.3\) is consistent with infinitely many distributions (one for each value of \(p\) up to 0.3), so no single calculation is possible. We therefore always test at the specific boundary value stated in \(H_0\).

How do I choose between a one-tailed and a two-tailed test?

The choice must come from the research question, decided before seeing the data. Ask: does the question specify a direction? "Has the proportion decreased?" → one-tailed lower. "Is the drug more effective?" → one-tailed upper. "Has the proportion changed?" → two-tailed. If you pick the direction after seeing that, say, the sample proportion is higher than \(p_0\) and then write \(H_1: p > p_0\), you have effectively halved the evidence needed to reject \(H_0\), which inflates the false-alarm rate.

Why do we say "do not reject \(H_0\)" rather than "accept \(H_0\)"?

A high p-value tells us the observed data are consistent with \(H_0\) — but the same data may also be consistent with many other hypotheses. Failing to find evidence against \(H_0\) is not the same as proving it is true. The phrase "do not reject" keeps the logic honest: we have simply failed to accumulate sufficient evidence against the null, not demonstrated that it is correct.

Module 4

Significance Levels

The significance level \(\alpha\) is the probability of rejecting \(H_0\) when it is actually true — i.e. the probability of a false alarm. Choosing \(\alpha\) is a judgement about how much risk of a false alarm is acceptable.

Common choices are \(\alpha = 0.05\) (5%) and \(\alpha = 0.01\) (1%). The choice should reflect the consequences of an incorrect rejection:

5% Level

Suitable when an incorrect rejection has moderate consequences. Used in most A-level problems unless stated otherwise. "There is a 1 in 20 chance of wrongly rejecting a true \(H_0\)."

1% Level

Used when the cost of a false alarm is high (e.g. recalling a medicine, scrapping a production line). Harder to reject \(H_0\), so requires stronger evidence.

With a discrete distribution like the Binomial, we often cannot achieve the exact significance level we want. The actual significance level (also called the actual size) is the probability of falling in the critical region under \(H_0\), which may be slightly below \(\alpha\).

Example: \(X \sim B(20, 0.3)\) under \(H_0\). Testing \(H_1: p < 0.3\) at 5%. We need the largest \(c\) such that \(P(X \leq c) \leq 0.05\).

\(P(X \leq 2) = 0.0355\) ✓ | \(P(X \leq 3) = 0.1071\) ✗

So the critical region is \(X \leq 2\), and the actual significance level is 3.55% (not exactly 5%).

Visualising the significance level

Adjust \(n\), \(p_0\) and \(\alpha\) to see the binomial distribution and where the critical region falls.

\(n\) =20

\(p_0\) =0.30

\(\alpha\) =

Think about it

Why can't we always achieve exactly 5%?

With a discrete distribution the probabilities come in distinct "jumps". Between \(P(X \leq 2)\) and \(P(X \leq 3)\) there is nothing in between — the cumulative probabilities jump straight from 3.55% to 10.71% in the example above. So no integer critical value gives exactly 5%. We must choose the largest critical region whose probability does not exceed \(\alpha\).

Does a smaller \(\alpha\) make the test "better"?

Not necessarily — it depends what you mean by "better". A smaller \(\alpha\) reduces the probability of a false alarm (Type I error), but it also makes it harder to detect a real effect (increases the Type II error probability). The two types of error trade off against each other for a fixed sample size. Increasing \(n\) is the way to reduce both simultaneously.

Module 5

One-Tailed Tests

A one-tailed test is used when, before seeing the data, we have a specific directional suspicion — either that \(p\) has decreased, or that it has increased.

Lower-tail test: \(H_1: p < p_0\)

We are looking for evidence that \(p\) is smaller than claimed. Extreme results are small values of \(X\). The p-value is the probability of getting \(X \leq x_{obs}\):

\[\text{p-value} = P(X \leq x_{obs}) \quad \text{where } X \sim B(n, p_0)\]

If \(\text{p-value} \leq \alpha\), reject \(H_0\).

Worked example: A coin is tossed 15 times. The manufacturer claims it is fair (\(p = 0.5\)). It lands heads only 3 times. Test at 5% whether there is evidence that the coin is biased towards tails.

\(H_0: p = 0.5\) \(H_1: p < 0.5\) \(X \sim B(15, 0.5)\) under \(H_0\).

\(P(X \leq 3) = P(X=0)+P(X=1)+P(X=2)+P(X=3)\)
\(= 0.00003 + 0.00046 + 0.00320 + 0.01389 = 0.01758\)

Since \(0.01758 < 0.05\), we reject \(H_0\). There is significant evidence at the 5% level that the coin is biased towards tails.

Resolving our opening example: Back in Module 1 we asked about a manufacturer's claim that 30% of sweets are red, when a bag of 20 sweets contained only 2 red. We can now answer it.

\(H_0: p = 0.3\) \(H_1: p < 0.3\) \(X \sim B(20, 0.3)\) under \(H_0\).

\(P(X \leq 2) = 0.0355\)

Since \(0.0355 < 0.05\), we reject \(H_0\). There is significant evidence at the 5% level that the true proportion of red sweets is less than 30% — this is unlikely to be just bad luck.

Upper-tail test: \(H_1: p > p_0\)

We are looking for evidence that \(p\) is larger than claimed. Extreme results are large values of \(X\). The p-value is the probability of getting \(X \geq x_{obs}\):

\[\text{p-value} = P(X \geq x_{obs}) = 1 - P(X \leq x_{obs} - 1)\]

Worked example: A drug trial claims 40% of patients improve without treatment. With the new drug, 14 out of 20 patients improve. Test at 5% whether the drug is effective.

\(H_0: p = 0.4\) \(H_1: p > 0.4\) \(X \sim B(20, 0.4)\) under \(H_0\).

\(P(X \geq 14) = 1 - P(X \leq 13) = 1 - 0.9935 = 0.0065\)

Since \(0.0065 < 0.05\), we reject \(H_0\). There is significant evidence at the 5% level that the drug increases the proportion who improve.

Common mistakes

For an upper-tail test, calculating \(P(X \leq x_{obs})\) instead of \(P(X \geq x_{obs})\)

The p-value must capture how extreme the observation is in the direction of \(H_1\). For \(H_1: p > p_0\), large \(X\) is the "surprising" direction, so we need the upper-tail probability \(P(X \geq x_{obs})\). Using the lower tail gives the probability of the opposite kind of extremeness — the answer will be wrong and will typically lead to the wrong conclusion.

Choosing the tail direction after seeing the data

The alternative hypothesis must be chosen based on the research question before looking at the data. If you pick the direction that makes your result look more significant after seeing it, you have effectively doubled your significance level. This is a serious statistical error sometimes called "p-hacking".

Module 6

Two-Tailed Tests

A two-tailed test is used when we simply want to know whether \(p\) has changed from \(p_0\), without a prior reason to expect a particular direction.

For \(H_1: p \neq p_0\), both very small and very large values of \(X\) are evidence against \(H_0\). We split the significance level equally between the two tails:

\[\text{Each tail uses } \frac{\alpha}{2}\]

For the observed value \(x_{obs}\), find which tail it falls in, then compare the tail probability with \(\dfrac{\alpha}{2}\):

If \(x_{obs}\) is small (below the mean)	If \(x_{obs}\) is large (above the mean)
p-value for this tail \(= P(X \leq x_{obs})\)	p-value for this tail \(= P(X \geq x_{obs})\)
Reject \(H_0\) if \(P(X \leq x_{obs}) \leq \dfrac{\alpha}{2}\)	Reject \(H_0\) if \(P(X \geq x_{obs}) \leq \dfrac{\alpha}{2}\)

Worked example: A spinner is claimed to give red with probability 0.25. In 30 spins it gives red 12 times. Test at 5% whether the probability has changed.

\(H_0: p = 0.25\) \(H_1: p \neq 0.25\) \(X \sim B(30, 0.25)\) under \(H_0\).

Mean under \(H_0\): \(np = 7.5\). Since \(12 > 7.5\), test the upper tail.

\(P(X \geq 12) = 1 - P(X \leq 11) = 1 - 0.9558 = 0.0442\)

Compare with \(\alpha/2 = 0.025\): \(0.0442 > 0.025\).

Do not reject \(H_0\). Insufficient evidence at the 5% level that the probability of red has changed.

Note on the two-tail p-value: An alternative convention is to double the smaller one-tail probability and compare with \(\alpha\) directly. Both methods are equivalent. At A-level, the "compare each tail with \(\alpha/2\)" approach is clearest.

Common mistakes

Comparing the tail probability with \(\alpha\) instead of \(\alpha/2\) in a two-tailed test

This is the most common error. In a two-tailed test the significance level is split between two tails, so each tail is tested at \(\alpha/2\). Comparing the one-tail probability with \(\alpha\) rather than \(\alpha/2\) is effectively performing a one-tailed test at \(\alpha\) — it doubles the probability of a false alarm.

Using both tails simultaneously (computing \(P(X \leq x)\) AND \(P(X \geq x)\))

You only need to work with the tail in which the observed value lies. If \(x_{obs}\) is above the mean, compute the upper tail probability and compare with \(\alpha/2\). If it is below the mean, compute the lower tail and compare with \(\alpha/2\). Adding both probabilities together is not the correct approach.

Module 7

Critical Regions

The critical region (or rejection region) is the set of values of the test statistic that lead to rejecting \(H_0\). Identifying the critical region in advance is an alternative — and often more efficient — approach to hypothesis testing.

Instead of computing a p-value each time, we can work out once which values of \(X\) would lead to rejection. Then for any observation we simply check: is \(x_{obs}\) in the critical region?

Finding the Critical Region: Lower Tail

Find the largest value \(c\) such that \(P(X \leq c) \leq \alpha\).

The critical region is \(\{0, 1, 2, \ldots, c\}\).

The actual significance level is \(P(X \leq c)\), which may be less than \(\alpha\).

Finding the Critical Region: Upper Tail

Find the smallest value \(c\) such that \(P(X \geq c) \leq \alpha\).

The critical region is \(\{c, c+1, \ldots, n\}\).

Finding the Critical Region: Two-Tailed

Find the largest \(c_L\) with \(P(X \leq c_L) \leq \alpha/2\). Lower critical region: \(X \leq c_L\).

Find the smallest \(c_U\) with \(P(X \geq c_U) \leq \alpha/2\). Upper critical region: \(X \geq c_U\).

The acceptance region is \(c_L < X < c_U\).

Example: \(X \sim B(25, 0.4)\) under \(H_0\), test \(H_1: p \neq 0.4\) at 5%.

Lower tail (\(\alpha/2 = 0.025\)): \(P(X \leq 5) = 0.0294 > 0.025\); \(P(X \leq 4) = 0.0095 \leq 0.025\). So \(c_L = 4\).

Upper tail (\(\alpha/2 = 0.025\)): \(P(X \geq 15) = 0.0344 > 0.025\); \(P(X \geq 16) = 0.0138 \leq 0.025\). So \(c_U = 16\).

Critical region: \(X \leq 4\) or \(X \geq 16\). Actual significance level: \(0.0095 + 0.0138 = 2.33\%\).

Critical Region Explorer

Set the parameters and test type to find the critical region automatically.

\(n\) =25

\(p_0\) =0.40

Think about it

What is the "acceptance region"?

The acceptance region is simply the complement of the critical region — the set of values for which we do not reject \(H_0\). If the test statistic falls in the acceptance region we have insufficient evidence to reject \(H_0\). The name is slightly misleading: we are not "accepting" that \(H_0\) is true, only failing to reject it.