Six rules of thumb for determining sample size and statistical power

This short guide from the Poverty Action Lab presents six rules very clearly, with helpful diagrams to explain or reinforce the points.

Four of them are relevant for planning quantitative data collection even if you’re not using a control group.

This resource and the following information was contributed by Patricia Rogers.

Authors and their affiliation

Poverty Action Lab

Key features

The guide presents six ‘rules of thumb’ that affect sample size and statistical power. Two of them relate specifically to random assignment (whether the treatment group and control group are the same size, and whether randomization is done at the level of individuals or larger units such as schools or villages) but the other four apply more generally to sampling in order to produce quantitative estimates.

• Rule of Thumb #1: A larger sample increases the statistical power of the evaluation.
• Rule of Thumb #2: If the effect size of a program is small, the evaluation needs a larger sample to achieve a given level of power.
• Rule of Thumb #3: An evaluation of a program with low take-up needs a larger sample.
• Rule of Thumb #4: If the underlying population has high variation in outcomes, the evaluation needs a larger sample.
• Rule of Thumb #5: For a given sample size, power is maximized when the sample is equally split between the treatment and control group.
• Rule of Thumb #6: For a given sample size, randomizing at the cluster level as opposed to the individual level reduces the power of the evaluation. The more similar the outcomes of individuals within clusters are, the larger the sample needs to be.

The rules are illustrated with useful diagrams to explain or reinforce the points.

How have you used or intend on using this resource?

I could imagine using this as part of the training materials for a skills development class in quantitative data collection and analysis. While the concepts would probably be included in a statistics textbook, the format of this guide is likely to be clearer.
The material could also be used to work with an evaluation advisory group or steering group to help them understand design choices and tradeoffs.

Why would you recommend it to other people?

It presents some important ideas very clearly in terms of the clear terminology, the diagrams and the uncluttered design.