In this new blog article we dig deeper into the use of the “don’t know” and “no opinion” (“DK/NO”) answer options in closed questions. After all, there is some debate in the scientific literature on whether to include or omit them. After extensively discussing the pros and cons of adding/omitting these answer options, we will provide you with some recommendations on how to deal with these answer options.

First things first. What do we mean by the “DK/NO” answer options? The idea behind them is very simple. It allows respondents to indicate that they do not know the answer to the question or do not have an opinion on a particular issue (see figure below).

1. Pros and Cons of adding the “DK/NO”

Now that we clarified what we understand by “DK/NO”, we can start discussing the pros and cons of this answer option.

One of the main advantages of adding a “DK/NO” is that it reduces noise in the data. In other words, if you would omit the “DK/NO”, respondents that do not know the answer to the question or respondents without a strong opinion would nonetheless be forced to choose an opinion (aka non-attitude reporting). This creates noise in the results. Adding a “DK/NO” reduces this noise.

However the idea of adding a “DK/NO” in order to minimise non-attitude reporting is based on two assumptions (Krosnick, et al., 2002). First of all, there are only two types of respondents, i.e. a) respondents that have opinions on any given issue and are aware of possessing those opinions and b) respondents that do not have opinions on any given issue and are equally aware of this. I.e. respondent with a fair amount of self-knowledge. Second, respondents are presumed to act rationally.

As a result, the first type of respondents are supposed to report their opinions, regardless of a “DK/NO” being included or omitted. The second type are then presumed to choose the “DK/NO” option when it is offered. As we already discussed, this type of respondents will most likely fabricate an opinion in order to appear opinionated when there isn’t a “DK/NO” offered. In order to avoid this fabrication, several survey researchers advocate for adding a “DK/NO”.

However, there appears to be a fundamental flaw in these assumptions. After all, it is assumed that a “DK/NO” only attracts those respondents that truly do not know the answer to the question or do not have an opinion on the surveyed topic. Yet, you can never be 100% certain that an opinionated respondent will not opt for the “DK/NO” option. As a result, the initial assumptions do not hold. That is why it is argued that you should not automatically presume that every time a respondent chooses for a “DK/NO”, (s)he cannot report a meaningful response.

So why would an opinionated respondent opt for the “DK/NO”, rather than expressing his/her true opinion? In the scientific literature various arguments are being put forward. First, respondents will opt for the “DK/NO” if they are not fully certain of the meaning of a question (e.g. Feick, 1989). Second, to avoid thinking and/or committing themselves (e.g. Oppenheim, 1992). Third, when the survey exceeds their motivation or their ability (Krosnick, 1991).

These final two arguments represent what Krosnick (1991) calls the ‘Theory of Survey Satisficing’. It is argued that answering attitudinal survey questions entails cognitive work for the respondents. It is not easy to answer these questions as respondents have to think about them. After all, respondents have to create a judgment on an attitudinal question and translate this judgment into a response (e.g. Schwarz & Bonner, 2001). When the amount of cognitive work exceeds the respondent’s motivation or ability, they will start looking for ways to avoid the workload while still appearing as if they are carrying on with the survey. A simple way of avoiding this cognitive workload in a survey is opting for the “DK/NO”.

That is why adherents of the ‘Theory of Survey Satisficing’ argue in favour of omitting the “DK/NO”. As such, so-called ‘satisificors’ do not have the opportunity to skip the hard work by choosing the “DK/NO”.

In recent years, scientific consensus has shifted towards omitting the “DK/NO”. For instance Krosnick et al. (2002) found that the inclusion of a “DK/NO” did not reliably improve the quality of the data. Instead it is possible that respondents attracted by the “DK/NO” would have provided substantive answers if there was not a “DK/NO” offered. The authors even conclude: “Analyses at the aggregate level suggest that many or even most respondents who choose an explicitly offered no-opinion response option may have meaningful attitudes, but we cannot rule out the possibility that some people do so because they truly do not have attitudes.”

2. Recommendations

Now that we learnt the pros and cons of the “DK/NO”, should you offer or omit this answer option?

As so often, it depends.

It depends on whether you can afford to waste some potential useful data. After all, gathering survey data is rather expensive. It requires a significant investment in time and money. As a result, following the ‘Theory of Survey Satisficing’, researchers that offer a “DK/NO” may be wasting potentially useful responses from people that otherwise would have opted for a substantial answer.

It depends on the nature of the questions and the profile of the respondents. It makes intuitively more sense to add a “DK/NO” for a factual question, rather than an attitudinal one. Ditto, if you are surveying an audience that has every possibility of not knowing the answer. For instance, if you are asking your customers to rate your helpdesk, it makes sense to add a “DK/NO” if you are not 100% sure that every customer has had contact with the helpdesk. However, it does not when you are surveying high school students on their attitudes towards bullying their fellow students.

It depends on whether you have a large enough sample as offering a “DK/NO” will most likely reduce your effective sample size. After all, a part of your respondents will opt for the “DK/NO” and, as a consequence, will most likely not be included in the analysis. The reduced impact in the data will be even stronger for certain socio-demographic groups. It has for instance been shown that lower educated respondents tend to opt for the “DK/NO” far more often than higher educated respondents.

Finally, if you decide to offer a “DK/NO”, you can try to get substantive data from respondents that opted for the “DK/NO” by asking them a follow-up question. You might for instance ask them whether they lean towards one of the substantive response options (Bradburn & Sudman, 1988).

So, how do you deal with the ‘don’t know/no opinion’ answer options in your surveys?

References

Bradburn, N. M. and Sudman, S. (1988), Polls and Surveys: Understanding What They Tell Us. San Francisco: Jossey-Bass.

Feick, L. F. (1989), “Latent Class Analysis of Survey Questions That Include Don’t Know Responses.”, Public Opinion Quarterly, 53: 525–47.

Krosnick, J. A., Holbrook, A. L., et al. (2002), “The impact of ‘no opinion’ response options on data quality. Non-attitude reduction or an invitation to satisfice?”, Public Opinion Quarterly, 66: 371-403.

Krosnick, J. A. (1991), “Response Strategies for Coping with the Cognitive Demands of Attitude Measures in Surveys.”, Applied Cognitive Psychology, 5: 213–36.

Oppenheim, A. N. (1992), Questionnaire Design, Interviewing, and Attitude Measurement. London: Pinter.

Schwarz, N. and Bohner G. (2001), “The Construction of Attitudes.” In Blackwell Handbook of Social Psychology: Intraindividual Processes, ed. Abraham Tesser and Norbert Schwarz, vol. 1, pp. 436–57. Oxford: Blackwell.