Some sage advice from Telegraph journalist Tom Chivers. For all of those who head out to collect interview data confident that the electronic data recorder in their pocket will meet their every need, always pack a notebook and pen, (better still two pens, even the most basic technology is fallible).
For some years as part of our undergraduate research methods teaching we have provided students with a mock-questionnaire which has been deliberately designed to enable them to identify the various potential flaws in survey design, and it is hoped, avoid them when they come to design their own surveys. As is often the case with such exercises the mock-survey has been carefully constructed to include a diverse range of errors and flaws, some obvious and some less so, but far more numerous than one would expect to find in any real survey one is likely to encounter.
Or maybe not. I recently encountered the above survey which is being distributed by the Conservative Party. I wondered at first if it was in fact a parody, someone’s attempt to create a deliberately flawed survey, or an unabashed attempt to make the Conservatives look bad, but it does appear to be genuine.
Leaving aside the politics, the flaws in the survey design are many and obvious, but it may, nevertheless, be instructive to draw some of them out.
- It is always a good idea to include some kind of introductory section explaining the purpose of your survey, particularly if it is an online survey. However, when doing so one must be careful to avoid doing this in such a way as to reveal to the reader in advance how you would like them to respond to your survey. Similarly, one should avoid associating particular responses with some variable independent of the survey, such as gender (‘men usually respond in this way…’) or political party. In this case the introduction clearly states that Conservatives support a particular position, if you are a Conservative supporter there really is no need to read the rest of the survey one need simply agree with everything that follows.
- When drafting survey questions be careful to avoid making causal links between unrelated phenomena, particularly if you are not going to provide evidence to support the link. So for example, asking people if they support controlling immigration is OK, but asking if they support it in order to ensure that ‘the economy and local services deliver’ suggests that the two are linked. If you are going to make such links in your questionnaire you really need to be able to back this up with data. It might be an idea to include some independent statistics at this point.
- Similarly, be sure to collect data on something that matters or is significant. Periodically students will want to carry out research on government planning for alien invasion or zombie attack, and while these are frightening prospects I usually point out that they are also extremely rare or perhaps non-existent phenomena, and they would therefore be better served focusing their research on something more significant. Similarly, while benefits and health tourism may be notionally possible I would want much more information about the scale of the problem before considering it worthwhile to include in a survey such as this. There is also a danger that if you start asking questions about a particular phenomena your research might actually serve to create public concern where it did not previously exist.
- Try to use clear and unambiguous language and avoid emotive terms or colloquialisms. For example, if you don’t read certain newspapers it may not be entirely clear what is meant by ‘benefits tourism’ or ‘health tourism’, or indeed whether these are good things or not. Similarly, there is a reference to ‘hardworking taxpayers’, who do sound like people I would like to support but I’m not sure that the tax-system, or anything else, discriminates between hardworking and non-hardworking taxpayers, it might be better to refer to ‘taxpayers’ or simply ‘people’. Conversely, I’m not sure what ‘bogus colleges’ are and I’m almost certain I’ve never seen one, but they sound like a bad thing, so how could one object to their closure? As long as its not my son’s college, but I’d never send my child to a bogus college, so closing them should be OK, shouldn’t it?
- If you are going to use a lot of questions which require a great deal of prior knowledge, particularly if you are not going to provide the respondent with some information to explain your questions, then it is often a good idea to include an additional check-box for ‘don’t know’ responses.
- Avoid generalising, while the questionnaire offers the reassuring suggestion that it is only bogus colleges which will be shut down, I am a little concerned with the suggestion that immigrants need to speak better English. There seems to be an assumption here that immigrants don’t speak very good English, but a large proportion of immigrants come from English-speaking countries like America, Australia or the Republic of Ireland. It may be better to ask which group of immigrants you would want to speak better English, perhaps breaking it down according to continent, then you could ask whether people think immigrants from Africa, Asia, Europe or America should be able to speak better English. On the other hand, it is possible that people might then misconstrue your meaning and assume you are being racist, so probably best to avoid this one altogether.
- It is also a good idea to frame your questions in such a way that all of the questions don’t elicit the same response. When completing surveys respondents often get bored and can easily slip into the habit of responding to every question in the same way, repeatedly ticking the yes or no box, particularly if responding to the first few questions in this way corresponds with their actual views. It is better then to change the wording of your questions and frame them in such a way that adopting one particular position does not lead you to respond to all of the questions in the same way. This not only keeps your respondent awake it also means they have to think, and as a result may offer a more considered response. Of course this does mean that they are more likely to spot any flaws in your survey, and may be less likely to give the response you desire.
- Above all avoid asking questions which are deliberately designed to prompt the respondent to answer in one way or another – these are known as leading questions. This can be done by making causal links between unrelated phenomena, using simplistic or emotive language, basing questions on certain assumptions, and framing questions so that each one prompts the same desired response. This is the cardinal flaw in survey design, it is important to remember at all times that you are conducting research to find out what people think and not to change the way people think, or in order to provide data which will support conclusions which you have already arrived at.
This is then a terrible questionnaire or an excellent example of a poor questionnaire. Any student submitting it as part of an assessment would fail, and it certainly would not be approved by any university research ethics process with which I am familiar. It is almost beyond parody, but then there is this.
This excellent post by Pat Thomson, Professor of Education at the University of Nottingham is essential reading. Professor Thomson, blogs about academic writing and research, and her posts are well worth following.
In this post Professor Thomson deals with the knotty and often ignored question of the difference between aims and objectives. She begins by noting that the problem of identifying a clear definition of aims and objectives is something for which dictionaries, and it might be added, research methods text books, are not particularly helpful.
She goes on to explain that aims relate to the overall intention of the research, what you hope to find out, show or illustrate. While objectives are the means by which you get there. Aims should be broad and there shouldn’t be too many of them, our research shouldn’t try to do too many things at once. Objectives should be a series of clearly defined steps which will enable us to achieve our aim.
It is important that research has both aims and objectives. Defining them is closely bound up with justifying our research in two important respects i) explaining why the research is worthwhile and needs to be carried out and ii) making it clear that we have some idea about how to go about it. It is no good identifying an aim if we have no idea how we are going to achieve it. Similarly, we shouldn’t carry out research without knowing what it is we want to know. That’s not the same as anticipating our results but does mean having some idea about what we want to find out about and why. Which brings us back to having a clear research question.
Confusion about this is not as uncommon as you might think. I have heard very experienced academics, who should know better, explain that the aim of their research is to collect or provide information about something. You might be very good at doing this, academics are often very good at collecting and indeed analysing data. But the important point is that all of that data and analysis should tell you something. The aim of your research should never simply be to collect information! We can easily forget this when we are in the midst of designing clever data collection instruments and frantically collecting data.
Imagine a group of researchers trying to secure funding for a large scientific research project. The objectives of their research involve crashing sub-atomic particles into each other at high speed. Nobody has done this before and they think they have come up with a sure-fire way of achieving it, which involves building a circular structure about 17 miles long, around which particles will be accelerated and then crashed into each other at high speed causing them to break up. Building such a structure and then operating it effectively will be a remarkable achievement. The methodology alone for such a project would fill several volumes. However, no matter how clever and novel the process might be, nobody would fund such research unless the researchers were able to explain the point of such activity. Those clever researchers at CERN are not firing particles at each other just for fun, nobody pays researchers just to have fun. The Large Hadron Collider is a very large and complex data collection instrument, which amongst other things aims to prove the existence of particles, the existence of which has hitherto solely been a theoretical proposition. It aims to do a whole lot of other things as well, but I’m no Brian Cox (something with which I’m confronted on an almost daily basis).
The LHC is a useful example of the relationship between aims and objectives in another respect, in that it illustrates that the aims of our research may not be defined by us. The theoretical proposition being tested at the LHC was first expounded by a group of physicists, including Peter Higgs, in 1964. Which is a timely reminder to read widely, not to dismiss work which was written before we were born, or before the internet, and that the most important part of research papers may the section at the end which identifies the questions the author was unable to answer. In the social sciences as well as the natural sciences, it is important to remember that there are many more questions out there than there are answers. Answering those questions should be our aim, how we go about it will define our objectives.
I digress, but in case you’ve forgotten where we came in, read Pat Thomson’s excellent practical guide to identifying your aims and objectives.
Students embarking on their undergraduate research project often claim that they want to carry out some form of discourse analysis without any clear idea what this means. This useful article from the PoliticsEastAsia blog provides a comprehensive introduction to discourse analysis with some useful links.
I don’t like the phrase, ‘library-based research’. It implies that research either takes place in the library or somewhere else. That students can choose between undertaking their own fieldwork or spending time in the library reading the findings of others in books and journal articles. In reality all research is library-based. A research project involving three months studying the mating habits of a little-known dung beetle in the Amazon rain-forest is likely to be preceded by a similar, although probably longer, period in the library seeking to amass what is already known about dung beetles and their mating habits. The phrase also implies that carrying out research in the library simply involves reading the work of others, and cannot therefore lead to the development of something new. This underestimates the way in which new findings, theoretical models and conceptual clarification, can emerge from challenging existing thinking, but also ignores the extent to which libraries are no longer just repositories of already published work (if they ever were), but now provide access to a wide range of information through advanced search engines, online databases, electronic archives as well as other archival and documentary collections.
All of which is largely a pretext designed to allow me to shoehorn into the blog a further extract from Kristin Luker’s Salsa Dancing into the Social Sciences. When conducting research in what she, rather inelegantly, refers to as ‘the age of info-glut’, Luker argues somewhat more eloquently, that librarians are essential guides whom researchers ignore at their peril:
Librarians, along with pediatricians, are among the greatest human beings in the universe. One of my colleagues at Berkeley calls them the “pit bulls of democracy” – as our government increasingly tries to hide things from us, librarians are among the few souls fighting back. They love the thrill of the chase as much or more than you do… If it was stupid thirty years ago to avoid reference librarians, it is downright suicidal now. Information has become a commodity – it is being bartered, sold, and arranged in more ways than anyone except a professional librarian can keep up with. Your job is to analyze information; a librarian’s job is to help you find it in the first place.
Information these days is like Europe after the fall of the Roman Empire. It’s entirely in disarray: every little principality mints its own money, passes its own laws, speaks its own language, and has its own rituals. Imagine trying to be a trader selling your wares as you go from town to town, each one under the sway of another prince (or princess) and his/her personally tailored ways of doing business… Only a reference librarian can teach you the tricks of the various systems and direct you to the ones you most need.
I often say, and I am not entirely joking, that you should court your reference librarian as you would court your future (or present) spouse. They are as overloaded as anyone else these days, all the more so being the pitbulls of democracy. So say “thank you” often. Write thank-you notes. Bring them coffee and cookies and chocolates. When they help you a lot, write a letter to the head of the library about how inventive, creative and helpful this person was. Always thank them in the “acknowledgements” section of your book or articles. Behind every good research project written by a salsa-dancing social scientist stands a great librarian, and maybe even a phalanx of them.
Kristin Luker (2008), Salsa Dancing into the Social Sciences: research in an age of info-glut, Harvard University Press, pp.85-6
Our very own pitbulls of democracy have just been awarded a customer service award as ‘a beacon of innovation and best practice, and an exemplar of excellent customer service.’ While we should perhaps gloss over the notion that library users are customers, and the guff about ‘the wider corporate University ethos’ this is nevertheless testament to a hard working and innovative library. I am constantly impressed by the extent to which the University library seeks to meet student needs with things like 24 hour opening, workshops and 1 to 1 training, the excellent (and oh so necessary) Harvard referencing guide and app, and the More Books scheme, which allows students at all levels to request new additions to stock. When students contact their Academic Subject Librarian, as they most certainly should, remember to thank them, but perhaps don’t refer to them as a pitbull, or indeed court them like a future spouse. That would be weird.