When we designed Survature’s survey interface, we envisioned building a highly engaging survey experience. What we found blew our minds. In the online survey research industry, completion rates is defined as the percentage of people who start and complete an online survey. For example, if 100 people started the survey and 50 completed the survey, it would have a 50% completion rate. Completion rates depend on a number of factors, like the length of survey, incentives for completing the survey, quality of the questions, intimacy of the community, etc.
In spite of these factors, Survature’s completion rate on every client survey thus far has resulted in 70-95% completion.
Why does a high completion rate matter? It matters for several important reasons. As a general rule of thumb and in reviewing the published literature, you need at least a 50% completion rate to conduct adequate analysis and reporting. What you do not want is what the industry calls the “silent majority” where more than half your targeted audience does not respond to your survey. In other words, a low completion rate means that the people who do respond (let’s say 30%) are unique in some way that will skew what should be the overall results.
For the purpose of this article, we will focus on one important reason: limiting non-response bias. Non-response bias is understood as the difference in observed results between those who respond and those who do not respond. Limiting non-response bias is important when surveying a limited sample size. For example, you are studying the opinions of every chief financial officer in the Fortune 100 companies about the state of the economy. The lower the completion rate, the higher the probability that the results do not accurately reflect the collective opinions of this targeted group.
Furthermore, it is important to reduce non-response bias when you want to study the opinions of subgroups within a limited sample size by demographics. Imagine you recruit 100 people who share the same experience to take an online survey about their experience. These 100 people are divided equally into five age groups: twenty people in their 20s, 30s, 40s, 50s, and 60s. A 100% completion rate would result in no bias as to each represented age group. Suppose only 50% of the respondents completed. What is the probability that exactly ten people from each age group completed the survey so that each group is represented equally? The answer is less probable than if 100% completed, than if 90% completed, and so forth.
Now suppose half of the people in each age group are male, and the other half are female. Suppose also that half the males are college educated, and the other half did not attend college. The same is true for the female group. Under this scenario, we have twenty resulting subgroups segmented by age, gender, and education. A 100% completion rate would mean that the opinions of each subgroup are represented equally. A lower completion rate would result in a higher probability that each subgroup is not represented equally.
In some studies, researchers correct for non-response bias by following up with non-responders through a telephone survey. The introduction of follow-on telephone surveys add more complexities and costs to the study. Thus, it is important to have a high completion rate to begin with.