Avoiding Bias
Recognize common research biases, distinguish systematic from random error, and use pilot testing to catch problems before they cost you a full study.
Bias is anything that pushes your results away from the truth in a consistent direction. Unlike random noise, it does not average out with a bigger sample. This page covers the biases most likely to affect online studies and the single most effective way to catch them: piloting.
Common Biases
| Bias | What it is | Mitigation |
|---|---|---|
| Confirmation bias | Designing or interpreting a study to favor what you already believe | Preregister; predefine analyses |
| Question-order bias | Earlier questions changing answers to later ones | Randomize or counterbalance order |
| Method bias | An artifact of how you measured, not the construct itself | Measure key variables more than one way |
| Social desirability bias | Participants answering to look good rather than honestly | Stress anonymity; normalize phrasing |
| Selection bias | The people who took part differ systematically from your target population | Define and report your sample; reason about who is missing |
Systematic vs Random Error
Every measurement carries error. The two kinds behave very differently.
- Random error is noise: unpredictable, scattered in both directions. It reduces precision but does not shift your average. More data and reliable instruments shrink it.
- Systematic error is bias: a consistent offset in one direction. It shifts your result regardless of sample size. You cannot fix it with more participants; you have to fix the source.
A bigger sample helps with random error and does nothing for systematic error. This is why design quality matters more than raw numbers.
Why Pilot Testing Matters
A pilot is a small run of your full study before you commit your budget. It is the cheapest insurance you can buy, and it catches problems that are invisible from the inside:
- Questions that participants read differently than you intended.
- Branching or display logic that sends people down the wrong path.
- Instructions that are unclear or attention checks that are unfair.
- Timing estimates that are wildly off, which affects fair pay.
- Data that does not export or merge the way you expected.
Running a Pilot on Terac
- Create your opportunity with a small participant cap, often 5 to 10 people.
- Consider targeting experienced participants for a cleaner technical read.
- Inspect the raw output end to end: does every field arrive, in the format you expect?
- Fix what you found, then launch the full study, excluding pilot participants if their data should not be combined.
Decide in advance whether pilot data will be included in your final dataset. If you change the instrument after piloting, the pilot responses are not comparable and should be excluded.
Fair Attention Checks
Attention checks reduce one kind of error but can introduce another if they are unfair. A fair check tests whether someone read instructions that genuinely matter for the task. An unfair check (ambiguous wording, trick phrasing) fails attentive participants and adds noise. Pilot your checks alongside the rest of the instrument, and treat a high failure rate as a signal that the check, not the participant, may be the problem.