Discover the Benefits of using Pure Questions in:  

Sign up to receive our e-mail newsletter

First name

Surname

E-mail address

* We won't pass on your details to anyone else

Question Design

Organisations often ask customers and employees questions using surveys and assessments. Improvements in communication technology, particularly the Internet, mean that this is now easier than ever.

However, designing the right questions is still very difficult. If questions aren’t asking the right things in the right way, then the answers provided (numerous or not) may be distorted, irrelevant and even misleading.

If you don’t ask the right questions, you won’t get the right answers.

Possible flaws in question design take many forms but here are a couple of examples:

Survey Question:

  1. On average, how many times a week are you unable to use your internet connection at home because it is too slow or not working at all?

    a. <1 (less than one)
    b. 1-5
    c. 6-10
    d. >10 (more than ten)

    This is poorly phrased and could generate misleading data. Can you tell why? Reveal.

    The first concern is that, in isolation, the response data from the question lacks meaning. With no indication of how often the respondent uses the Internet, knowing the frequency their connection fails tells us very little about their true quality of service. Use of the internet is also becoming more diverse in terms of bandwidth requirements. While a minor service disruption may not affect people reading web-sites, those streaming live video content or gaming in real-time would often be less fortunate.

    Even if this contextual data were gained from other questions, the wording is still worryingly subjective. For example, judgement of connectivity that is ‘too slow’ will depend on the individual. It will also depend on what they are doing at the time and how urgently they need to do it. There will also be differences in how quickly and often people will retry a broken connection. Would they count all of these attempts when answering the question above? Without clarification, the answers provided to this question could easily over or understate a problem and, worse, lead management to make poor decisions as a result.

Assessment Question:

  1. One of the reactor coolant pumps is operating 6 degrees Celsius above recommended operating temperature. This would be indicated in the control room by an alarm and an...

    a. ...red status light on the main system dashboard
    b. ...flashing red status light on the cooling system dashboard
    c. ...amber status light on the cooling system dashboard
    d. ...flashing amber status light on the cooling system dashboard

    Worryingly, you don’t need to know anything about the warning systems in order to answer correctly. Did you spot the grammatical clue? If not, or to check, please reveal

    The correct answer is C. There are a number of errors in the wording but the most significant is that the question stem ends with the word ‘an’. This suggests that any word following it will begin with a vowel (or an ‘h’). As all of the other answer options start with a consonant, C is much more likely to be the correct response.

    The objective of this question is to measure knowledge of the safety controls. However, given this error, it also measures knowledge of English grammar or ‘test-smartness’ (an ability to make an educated guess). It is also culturally biased as a native English speaker would be more likely to guess correctly than someone less proficient. This would happen even if they were as knowledgeable about the safety systems or indeed any other intended subject.

    Worst of all, this type of question error can create a dangerous illusion of knowledge. This poses an operational risk greater than not testing at all.

These wording mistakes are relatively easy to spot but, unfortunately, most are far more subtle. An additional worry is that so many questionnaires and assessments are now delivered anonymously and remotely by computers. This means that respondents can’t easily clarify questions or highlight ambiguities making errors even harder to detect.

Frustrated Participants

Unclear, leading or irrelevant questions are also frustrating for respondents. They immediately reduce the perceived and actual value of the process. Subsequently, levels of participation drop and support and credibility of any findings is diminished.

To find out more about avoiding or fixing these issues please read Improving Surveys or Improving Assessments

Alternatively, if you have any questions, or would like to discuss your requirements, please Contact us.