Eating one’s own dog food is a meme CheckMarket takes to heart. For example, we recently conducted a survey among our project managers and support staff. We asked them which survey mistakes they encounter the most often during a survey project. The results are broken down into 5 categories of common survey mistakes.

1. Not thinking about the analysis before starting the survey

“It’s about the results” isn’t just an aphorism; it is the secret of a successful survey project. A good questionnaire starts by asking yourself the right question: What do I want to find out? Don’t just say “I want to do a customer satisfaction survey”. Start by writing down specifically what you want to know from your respondents and ask just the questions needed to find out. We often see unnecessarily long surveys.

Once you have the right questions, think about how you will analyze them. For instance, if you would like to see a breakdown of the results for certain age groups, ask your respondents to which age group they belong and don’t ask them for their year of birth or their age in an open question. This will save you a lot of data cleaning afterwards.

2. Writing sloppy questions

When looking at the responses from our internal survey, a lot of the mistakes mentioned center around ‘Sloppy questions’. We are talking about basic things like:

• spelling mistakes
• no ‘N/A’ or ‘other’ options
• important questions that are not required
• no validations

We also see more serious mistakes that can invalidate results.

• Ambiguous questions

This means question wording that is unclear and open to interpretation.

For example, the question:
Would you be willing to relocate for a better job?

This might seem clear at first. However, the respondent doesn’t know exactly what relocation means (i.e. work somewhere else or live somewhere else? And where, in a different city or maybe country?)

Another example:
Please rank your satisfaction with company X on a scale of 1 to 5:

Does a ranking of 1 or 5 indicate the most satisfied?

• Wrong question types

Using checkboxes (multiple select) instead of radio buttons (single select). One lets the respondent select more than one response for the question. The other only lets the respondent select one response.

• Reversing scales

This means presenting the respondent with a list of questions scaled from 1 to 5 and having 1 be best some of the time and 5 be best some of the time. This is more controversial, some say this is actually a good way to keep respondents awake and to prevent indiscriminate answer patterns (i.e. giving the same answer all the way down the survey). We disagree; in our experience changing the scales around in a survey causes problems. Respondents tend to check the scale once and then apply it through the rest of the survey.

Here is an example of a rating scale that removes the scale direction issue completely:

3. Rushing it out there

It is amazing how often we see users work all week on their B-to-B survey, only to see them finally finish on Friday at 5 p.m. and then launch and send out their survey. Big mistake. Not only is the timing bad, but they forgot one of the most important steps, testing the survey.

Send the survey to a few colleagues. If you are going to use e-mail invitations, send your colleagues a test invitation before launching your survey. Did they notice it? Did it arrive in their SPAM folder? Ask for feedback. You also need to test the survey with your target audience. Ask a few members of your target audience if they would be willing to participate in a trial run of the survey. They may notice things you did not, such as acronyms or industry words that they are not familiar with.

Once you have tested your survey and are ready to launch it, carefully select the correct day of the week and time. Is your survey X2B, X2C or X2E? The X can stand for (B)usiness or (G)overnment.

X2B: Tuesday and Thursday in the morning between 9am at 11am. Never on Wednesday or Friday afternoon.

X2C: Which day seems to matter less than for X2B. The quality of the mail plays a more important role and incentives play a much bigger role.

X2E: Tuesday and Thursday in the morning between 9am at 11am or between 3pm and 4pm. Never on Wednesday or Friday afternoon.

If ‘Location’ is the mantra of real estate, then ‘Delivery’ should be the mantra of conducting surveys. The quality of the delivery determines the response rate of the survey, period. Pay attention to the 3 second rule.

• Recipient reads From line – the sender’s name –> next or delete
  Make the sender’s name recognizable to the recipient.
• Recipient reads Subject line –> next or delete
  Choose your subject line carefully in order to trigger the recipient. We often see a subject line like ‘Your opinion counts!’. Try something like ‘‘[FIRST NAME], What do you think we could do better?” Use variables from your address book .
• Recipient scans mail content –> click or delete
  Make sure your mail is branded, short and contains a call to action. The e-mail must be in the look of your website in order to get the highest response. If there is an incentive, place an image of it in the e-mail. If possible include a contact person in the mail as this dramatically increases the credibility of the mail.

After all those steps, the recipient arrives on the landing page of the survey. Keep this page short and include a simple easy multiple choice question.

5. Not doing anything with the results

Conducting a survey is just one part of the process. Once the field work of a survey is complete, it is time to analyze the data. But how to start? Well, if you followed our advice and started the project by writing down what you wanted to learn from the survey, use that as a framework for analyzing the data. Once you have that report, formulate a plan of action based on the results and then execute that plan. The last step of the plan is to schedule a follow-up survey to check if the goals of the action plan have been met.

Be sure to communicate the results or action plan back to the respondents of the survey if possible. This will increase the likelihood that they will participate in future surveys. People have a long memory. If they give their time to you by filling out your survey and they get nothing in return, they will remember that the next time you come knocking with a new survey. “There they are again, last time I told them what I thought and nothing has changed.” This is especially important for employee and client satisfaction surveys.

There you have them, 5 common survey mistakes we see and how to avoid them.

If you have any questions about this article or your survey project, please drop us a mail at [email protected].