Designing a survey — question types and sample size in plain English

Most survey errors happen before the first response arrives. The questions are leading, the scale is mismatched to the construct, or the sample is too small to answer the question you actually asked. Fixing any of these after the fact is impossible. This guide covers the survey design moves we teach in Research Goal's methodology cohort — written for working researchers, not statisticians.

Question types that don't bias your data#

Four question types cover 95% of survey research. Knowing when each one fits — and when it doesn't — prevents most measurement errors.

1. Likert scales — for attitudes#

Five-point or seven-point. Use five when you don't need fine discrimination; seven when you do. Never use even-numbered scales unless you specifically want to force a position (no neutral middle). Label every point in words, not just the endpoints — "agree" and "strongly agree" mean different things to different respondents.

2. Multiple choice — for categorical facts#

Always include "Other (please specify)" unless you've enumerated every possibility (e.g., yes/no, country of residence with full list). Always include "Prefer not to say" for sensitive demographics. Mutually exclusive AND collectively exhaustive — or your data has gaps.

3. Open-ended — for mechanism and language#

Use sparingly. One or two per survey at most. They earn you the language respondents use to talk about your topic — useful for follow-up qualitative work — but most respondents skip them or give one-word answers. Place them at the end, not the beginning.

4. Ranking — for relative preference#

Useful when you genuinely need ordinal data ("rank these five priorities"). Rarely useful past five items — ranking 10 things is cognitively expensive and noisy. Often better replaced by separate Likert scales for each item.

A sample-size rule that works#

Power calculations are correct but most researchers don't do them. A working rule that lands you in the right ballpark, without statistical software:

• For descriptive surveys (means, proportions) — 100 respondents per subgroup you want to characterise separately. Want to break out men vs women vs non-binary? 300 minimum.
• For correlations — 200 respondents to detect a medium correlation (r ≈ 0.2) at 80% power. 400 to detect a smaller one (r ≈ 0.15).
• For regression with k predictors — 50 + 8k as the minimum. Five predictors → 90 respondents. Ten predictors → 130. More is better.
• For comparing two means — 64 per group to detect a medium effect (d ≈ 0.5) at 80% power. 100 per group is a safer floor.

These are minimums. If your hypothesis depends on a small effect or you want robust subgroup analysis, multiply by 2 or 3. The cost of an underpowered survey is the entire study; the cost of an overpowered one is some extra recruiting.

Order effects#

The order of your questions changes the answers — sometimes dramatically. Two patterns to watch:

• Priming — asking about climate change before asking about energy policy shifts the policy answers measurably
• Anchoring — the first scale point a respondent sees calibrates how they use the rest of the scale
• Fatigue — answers to questions in positions 30+ are less reliable than those at positions 1–10
• Demographics at the end — never lead with demographics; they trigger stereotype-threat patterns

The pilot test you can't skip#

Run the full survey on 10–15 people before you send it to your real sample. Sit with three of them while they complete it. Ask: "Was any question confusing? Did any of them feel leading? Did you skip anything?" Every survey we've reviewed has at least one question that doesn't mean what the researcher thought it meant. The pilot catches it; the real sample tells you only that the question got noisy responses.

Wrapping up#

Four question types, a sample-size floor matched to your design, an awareness of order effects, and a real pilot with three people sitting next to you. Skip any of these and your data carries errors you'll never untangle. Do them and the analysis writes itself.