A pilot test is a small trial run of your survey with people like your real respondents, done before you send it out widely. It shows whether questions are clear, whether the skip logic works, how long the survey takes, and whether the data will export in a usable format. Fixing a problem in a pilot costs a few hours. Fixing it after fieldwork can mean discarding responses or repeating the study.
What a pilot test checks
A good pilot looks at six things:
- Wording. Do people understand each question the way you intended?
- Order and flow. Does the survey move logically from one topic to the next?
- Logic. Do people see the questions they should, and skip the ones that do not apply?
- Timing. How long does the survey really take, and where do people stop?
- Experience. Does it work on a phone, on a slow connection, and in the browser people will use?
- Data. Can you export the responses in the format your analysis needs?
Pilot test or pretest?
The two terms are often used loosely. A pretest usually means a quick review of wording, often with a few colleagues or a handful of target respondents talking through each question. A pilot is a fuller trial: the survey runs as it would in the real study, with a small group of respondents and the same tool, link, and instructions. Both are useful. Start with a pretest to catch wording problems, then run a pilot to test the whole experience.
Step 1: Set what the pilot must answer
Write down the questions the pilot needs to answer. For example:
- Do people understand the terms in questions 4 to 9?
- Does the screener remove the right people?
- Does the survey take less than 10 minutes on a phone?
- Does the export include every question, with clear column names?
A pilot with no specific questions tends to produce vague feedback such as “looks fine,” which does not help you change anything.
Step 2: Choose the pilot group
Pick people who match your target audience. If your main study uses a consumer panel of people who buy running shoes, pilot with people who fit that profile. Where possible, keep pilot respondents out of the final sample. If they have already seen the questions, their later answers may be affected.
Step 3: Run it like the real survey
Use the same tool, device types, invitation, incentive, and instructions you plan to use in the main study. A pilot run through a different link or on a different device will miss problems that the real study would hit.
Step 4: Collect feedback as well as answers
Numbers show what happened. Feedback shows why. Two methods work well:
- Think-aloud interviews. Ask a few people to answer the survey while they talk through their thinking. Ask them to explain what they think a question means. This is often called cognitive interviewing. A handful of interviews usually reveals the most important wording problems.
- Follow-up questions at the end. Add a short comment box, or ask: “Was any question hard to understand?” and “Was any answer option missing?”
Pay close attention to “Other” answers in the pilot. If many people write in the same answer, that option should probably be on the list.
Step 5: Check the data
Once the pilot responses are in, look at the data before you change anything.
Completion time and drop-off
Record how long each respondent takes. Compare the median time with your target. Look for the question where most people leave. A sudden drop-off often signals a confusing or sensitive question, or a page that loads slowly on a phone.
Skipped questions and straight-lining
Count the blanks for each question. A question that many people skip may be unclear, sensitive, or not relevant to them. Also check for straight-lining, where someone gives the same answer to every item in a grid. A few cases may be genuine. Many cases from the same pilot suggest the grid is too long or the questions are tiring to read.
Logic, screeners, and quotas
Test every path through the survey. Check that people who should be sent to a later section are sent there, and that people who should be screened out are removed. Confirm that quotas fill as you expect. The screener question guide covers how to write screener questions that are easy to answer and easy to check.
Export format
Export the pilot data and open it in the program you will use for analysis. Check that every question has a column, that the column names are clear, and that answer codes match what you expect. Problems that appear here are easy to fix before launch and much harder to fix afterward. If you are still choosing a tool, the guide to online survey tool types explains what to look for in export options.
Reliability checks for multi-item scales
If you use several questions to measure one idea, such as a set of statements about trust in a brand, you can check whether they hang together with an internal consistency measure such as Cronbach’s alpha. Treat the result with caution. Estimates from a small pilot sample are unstable, so use them only to spot obvious problems, such as one item that does not fit the others at all. Do not treat a pilot’s alpha as a final measure of quality. The Likert scale guide explains how to set up the scales themselves.
Pilot checklist
| Area | Check | Done |
|---|---|---|
| Wording | Respondents explain each question in their own words as intended | |
| Wording | No question assumes an answer (see the leading questions guide) | |
| Flow | Topics follow in a logical order | |
| Logic | Skip patterns send each person to the right questions | |
| Screener | The right people qualify, and the wrong people are removed | |
| Timing | Median completion time matches the target | |
| Drop-off | No single question causes a large share of people to leave | |
| Mobile | The survey works on a phone | |
| Export | The data opens cleanly, with clear column names | |
| Changes | Every change is recorded, with the reason |
Common mistakes
- Piloting with colleagues only. People who know the study will read questions differently from your target respondents.
- Changing things without checking the data. Make changes based on evidence, not just on personal preference.
- Piloting a different version. Test the exact survey you plan to send, including the final logic and invitation.
- Skipping the export test. A survey that looks perfect on screen can still produce a messy file.
- Treating a small pilot as proof. A pilot finds problems. It does not confirm your results.
- Skipping the pilot because the deadline is close. A short pilot, run over two or three days, still catches the errors that matter most.
A pilot is the last check before you spend money on fieldwork. For the wider design of the study, start with the market research survey questions guide.