For some years as part of our undergraduate research methods teaching we have provided students with a mock-questionnaire which has been deliberately designed to enable them to identify the various potential flaws in survey design, and it is hoped, avoid them when they come to design their own surveys. As is often the case with such exercises the mock-survey has been carefully constructed to include a diverse range of errors and flaws, some obvious and some less so, but far more numerous than one would expect to find in any real survey one is likely to encounter.
Or maybe not. I recently encountered the above survey which is being distributed by the Conservative Party. I wondered at first if it was in fact a parody, someone’s attempt to create a deliberately flawed survey, or an unabashed attempt to make the Conservatives look bad, but it does appear to be genuine.
Leaving aside the politics, the flaws in the survey design are many and obvious, but it may, nevertheless, be instructive to draw some of them out.
- It is always a good idea to include some kind of introductory section explaining the purpose of your survey, particularly if it is an online survey. However, when doing so one must be careful to avoid doing this in such a way as to reveal to the reader in advance how you would like them to respond to your survey. Similarly, one should avoid associating particular responses with some variable independent of the survey, such as gender (‘men usually respond in this way…’) or political party. In this case the introduction clearly states that Conservatives support a particular position, if you are a Conservative supporter there really is no need to read the rest of the survey one need simply agree with everything that follows.
- When drafting survey questions be careful to avoid making causal links between unrelated phenomena, particularly if you are not going to provide evidence to support the link. So for example, asking people if they support controlling immigration is OK, but asking if they support it in order to ensure that ‘the economy and local services deliver’ suggests that the two are linked. If you are going to make such links in your questionnaire you really need to be able to back this up with data. It might be an idea to include some independent statistics at this point.
- Similarly, be sure to collect data on something that matters or is significant. Periodically students will want to carry out research on government planning for alien invasion or zombie attack, and while these are frightening prospects I usually point out that they are also extremely rare or perhaps non-existent phenomena, and they would therefore be better served focusing their research on something more significant. Similarly, while benefits and health tourism may be notionally possible I would want much more information about the scale of the problem before considering it worthwhile to include in a survey such as this. There is also a danger that if you start asking questions about a particular phenomena your research might actually serve to create public concern where it did not previously exist.
- Try to use clear and unambiguous language and avoid emotive terms or colloquialisms. For example, if you don’t read certain newspapers it may not be entirely clear what is meant by ‘benefits tourism’ or ‘health tourism’, or indeed whether these are good things or not. Similarly, there is a reference to ‘hardworking taxpayers’, who do sound like people I would like to support but I’m not sure that the tax-system, or anything else, discriminates between hardworking and non-hardworking taxpayers, it might be better to refer to ‘taxpayers’ or simply ‘people’. Conversely, I’m not sure what ‘bogus colleges’ are and I’m almost certain I’ve never seen one, but they sound like a bad thing, so how could one object to their closure? As long as its not my son’s college, but I’d never send my child to a bogus college, so closing them should be OK, shouldn’t it?
- If you are going to use a lot of questions which require a great deal of prior knowledge, particularly if you are not going to provide the respondent with some information to explain your questions, then it is often a good idea to include an additional check-box for ‘don’t know’ responses.
- Avoid generalising, while the questionnaire offers the reassuring suggestion that it is only bogus colleges which will be shut down, I am a little concerned with the suggestion that immigrants need to speak better English. There seems to be an assumption here that immigrants don’t speak very good English, but a large proportion of immigrants come from English-speaking countries like America, Australia or the Republic of Ireland. It may be better to ask which group of immigrants you would want to speak better English, perhaps breaking it down according to continent, then you could ask whether people think immigrants from Africa, Asia, Europe or America should be able to speak better English. On the other hand, it is possible that people might then misconstrue your meaning and assume you are being racist, so probably best to avoid this one altogether.
- It is also a good idea to frame your questions in such a way that all of the questions don’t elicit the same response. When completing surveys respondents often get bored and can easily slip into the habit of responding to every question in the same way, repeatedly ticking the yes or no box, particularly if responding to the first few questions in this way corresponds with their actual views. It is better then to change the wording of your questions and frame them in such a way that adopting one particular position does not lead you to respond to all of the questions in the same way. This not only keeps your respondent awake it also means they have to think, and as a result may offer a more considered response. Of course this does mean that they are more likely to spot any flaws in your survey, and may be less likely to give the response you desire.
- Above all avoid asking questions which are deliberately designed to prompt the respondent to answer in one way or another – these are known as leading questions. This can be done by making causal links between unrelated phenomena, using simplistic or emotive language, basing questions on certain assumptions, and framing questions so that each one prompts the same desired response. This is the cardinal flaw in survey design, it is important to remember at all times that you are conducting research to find out what people think and not to change the way people think, or in order to provide data which will support conclusions which you have already arrived at.
This is then a terrible questionnaire or an excellent example of a poor questionnaire. Any student submitting it as part of an assessment would fail, and it certainly would not be approved by any university research ethics process with which I am familiar. It is almost beyond parody, but then there is this.