Sloppy Technique Skews Online Survey Accuracy
Posted on | February 18, 2011 | No Comments
Today’s web-based survey software has put easy-to-use research tools into the hands of more marketers than ever before. Even creating sophisticated surveys – with skip logic or matrix questions – can be relatively easy to accomplish. But not all users implement best practices with the same rigor professional researchers employ. Sloppy survey techniques can skew survey results. Any decisions made or actions taken based on those results can jeopardize business plans and outcomes.
To help emarketers achieve sound results, we thought we’d address two basic survey concepts that are often misunderstood or simply ignored in online marketing survey design: population and sample size. Sometimes used interchangeably, these terms mean different things. Getting one or the other wrong can invalidate survey results.
Population and sample size aren’t complex concepts. Think about “population” as the universe of people who meet the basic qualifications for answering a particular survey. This could be a large universe, such as all US citizens for a US census, or it could be much smaller, such as Cleveland businesses with less than 50 employees for a regional US Chamber of Commerce questionnaire on health insurance. In contrast, “samples” are “statistically significant” subsets of just such populations.
The key to designing a survey that reflects results that are true for a total population is to make sure, first, that you’ve correctly identified the group you want to survey and, second, that the sample you use for your survey is sized appropriately. In general, the larger the sample size, the better the chances that your results will be “statistically significant” for the total population.
A couple of concepts to keep in mind:
- Margin of error: The margin of error is a statistic expressing the amount of random sampling error in a survey’s results (ref: Wikipedia). The number is generally reported as a plus or minus percentage. If, for example, your margin of error is 5%, then for any given answer you have a chance of a five point discrepancy higher or lower.
- Confidence level: Also expressed as a percentage, the confidence level tells you how sure you can be that a given percentage of the entire population would pick an answer within the margin of error. A 95% confidence level – which most researchers find adequate – means you can be 95% certain; the 50% confidence level means you can be much less certain.
So, how do you figure out what size sample you need? Well, here’s the good news: there are lots of sites on the Internet that will do the computation for you and Ennect itself provides a random selection capability right within our survey tool. However, if you want to do the selection yourself, we’ve found several outside sites that have relatively easy-to-use calculators. Here’s some to try:
- http://www.custominsight.com/articles/random-sample-calculator.asp — Custom Insight
- http://www.researchinfo.com/docs/calculators/samplesize.cfm — Creative Research Systems
- http://www.americanresearchgroup.com/sams.html – American Research Group
XC6CDZR5EREJ
Comments
Leave a Reply