Hypothesis Tests, P Values & Power
Interpret statistical evidence alongside effect size, uncertainty and the decisions made before analysis.
Quick revision ↓ Medical Dictionary
01Null hypotheses and p values
A null hypothesis commonly specifies no difference or no association. A p value is the probability, assuming that hypothesis and the statistical model, of a result at least as incompatible with it as the observed result. It is not the probability that the null is true, the probability that findings occurred by chance alone, or a measure of the clinical importance of an effect.
02Error types and power
A type I error rejects a true null hypothesis; its long-run probability is controlled by the chosen significance level under the testing procedure. A type II error fails to reject a false null for a specified alternative. Power is one minus that type II error probability. Power calculations require assumptions about a target effect, variability or event rate, significance level and design, not merely a desired sample size.
03Clinical versus statistical importance
A very large study may detect an effect too small to matter clinically, while a small study may have an important estimate with wide uncertainty. Report the effect and interval, and compare them with a justified clinically important difference. A non-significant superiority test does not prove equivalence. Equivalence and non-inferiority require appropriately designed studies, prespecified margins and analyses suited to those questions.
04Multiplicity and prespecification
Testing many outcomes, time points and subgroups increases opportunities for false-positive findings if multiplicity is ignored. Prespecify the primary question and analysis, distinguish confirmatory from exploratory findings and use appropriate adjustments when needed. An unexpected subgroup result can generate a hypothesis, but it should not be portrayed as established simply because its p value happens to cross a conventional threshold.
05Sample size and transparent analysis
Plan sample size before collecting data, allowing for expected missingness, clustering and the chosen outcome. Report assumptions and deviations. After completion, confidence intervals are usually more informative about uncertainty than post hoc power computed from the observed effect. Avoid repeatedly checking results and stopping opportunistically without a valid sequential design, because this changes the error properties of an ordinary fixed-sample test.
High-Yield Revision
- A p value does not measure effect size or clinical importance.
- Power refers to a specified alternative effect and design.
- Non-significance is not proof of equivalence.
Common Pitfalls
- Do not select the primary outcome after seeing which result is significant.
- Do not interpret a p value as the probability that a hypothesis is true.
References & Further Reading
Use the full methods and relevant chapters when appraising a paper. These sources support further study; the examples and questions above are original teaching exercises.
- Oxford CEBM — Critical appraisal tools
- Cochrane Handbook — Methods for evidence synthesis
- CDC — Analysing and interpreting epidemiological data
- ASA — Statement on p-values and statistical significance
- Oxford CEBM — Likelihood ratios
- Cochrane — GRADE and summary of findings
- WMA — Declaration of Helsinki, 2024 revision
- EQUATOR — Reporting guideline library
- NIH OBSSR — Mixed-methods research resources
- Institute for Healthcare Improvement — Model for Improvement