Beware the alibi of photocopies

IMG_2694There are many intellectual aspirins in Oliver Burkeman’s weekly, ‘This column will save your life’, in the Saturday edition of The Guardian. The column ‘What unread books can teach us’ is particularly pertinent to the budding researcher.

In it Burkeman, drawing on observations by the writers Umberto Eco and Nassim Taleb, defends the tendency of writers, and indeed academics, to surround themselves with groaning shelves of books, many of which they will never read. While some assume that this is simply showing off, Umberto Eco points out that books are a research tool, and like any good toolbox our shelves should be filled not only with those we will use on a daily basis, but others which may serve very particular but no less important function at some point in the future. Similarly Taleb argues that the real aim of a personal library should be to fill it with the books you have not read, rather than those you have. According to Burkeman, Taleb advises that your bookshelves should contain ‘as much of what you do not know’ as finances allow. There is a simple logic to this. If we assume that there is no way that we will ever be able to read everything, then there is always more to be learnt from the books that we haven’t read than from the relatively small number that we have. As Burkeman observes ‘unread books are where the action is.’

However, Burkeman’s column also contains a warning from Eco that we should not assume that accumulating unread work is the same as accumulating knowledge. Eco refers to this as ‘the alibi of photocopies’. By this Eco identified a practice which is surely familiar to all of us. That by the simple act of photocopying or printing a book chapter or journal article we sometimes, perhaps often, assume that we have read it and internalised it.

It is important to remember that accumulating data, whether that is books, articles or research data, is not the same as having read or analysed it. The pile of books and journal articles sitting on the corner of your desk does not constitute research. Research is what begins to happen when you read them. Moreover, as Burkeman points out, Eco was writing about the alibi of photocopies in the mid-1970s. Developments in information technology have significantly expanded our capacity to accumulate unread texts, but have had little impact on our capacity to read them. There is then real value in unread books, as long as we don’t make the mistake of assuming that we’ve read them.

Posted in Uncategorized | Tagged , | 1 Comment

Writers on writing: Ian Rankin

On the face of it there may be little similarity between the social sciences and crime fiction, but I often find that there is a great deal to be learnt from professional novelists. The one thing we do have in common is the need to communicate ideas through writing. One of the reasons why successful novelists are successful is their ability to write and in many cases to write quickly. Moreover, one often finds that successful writers have given considerable thought to their craft, to the process of writing, something which data-obsessed social scientists perhaps do less of. In this context I particularly liked this interview with one of my favourite novelists, the Scottish crime writer, Ian Rankin, in The Globe and Mail in which he outlines five habits which can lead to successful writing. Not all of these are applicable to research in the social sciences. In my own writing I have yet to encounter the knotty problem of writing a sex scene, and in non-fiction writing the ending is rarely much of a surprise. Nevertheless, Rankin’s advice about the importance of getting into the habit of writing, is a lesson from which we could all benefit:

Just keep writing

If I’m working on a novel, I try to write every single day. It may not be much, but it’s seven days a week. I think that by writing quickly you inject pace. If a book is written quickly, it tends to be a quick read as well. My first draft will be very rough, but it won’t take me much more than 30 or 40 days. I remember one writer that I met couldn’t start chapter two until he got chapter one just right. That’s just not how I work. My rule is to just get the thing down on paper. Even if there are mistakes, misjudgments, I’ll ignore that and know that I can go back later. I remember I was talking with an agent one time. He asked how things were going and I said they were going great, but that I was just about to take a break because I have to research this one thing. He said no, no – just make a note to yourself to do the research later and keep writing. I save a lot of time these days by doing the bulk of the research between the first and the second draft. Research can be a rabbit hole and you don’t come out for weeks. Once you have a first draft, you know what you need and that saves a lot of time.

It is important to remember, as Rankin acknowledges, that there are different approaches to writing and that some people would baulk at the notion of writing a quick first draft before the research is complete. But it is hard to dismiss the notion that when one has some momentum it is probably better to carry on writing than to stop to check every fact, as long as one remembers to check them later. As someone who can spend a great deal of time chasing ever more data, Rankin’s characterisation of research as ‘a rabbit hole… you don’t come out of for weeks’ is particularly striking. We’re not writing fiction so there are times when we can’t just make it up, but there are also times when we all need to avoid the rabbit holes and just keep writing.

Posted in Uncategorized | Tagged , | Leave a comment

On the importance of literature reviews

In a recent article in The Washington Post, Daniel W. Drezner, professor of international politics at the Fletcher School of Law and Diplomacy at Tufts University, neatly summarised the importance of literature reviews in academic papers:

Done well, a literature review states what the extant research has to say about a topic. Is there a scholarly consensus on the question or not? Are there contending schools of thought? What puzzles persist? What data controversies are there?

Peer referees like to see literature reviews in papers, because it sends a signal that the author is keenly aware of what preceded his or her article. An inadequate literature review can be the kiss of death to a paper if the author then proposes an argument or test that the referee knows about but goes uncited.

More importantly, a literature review is the way for someone who is not an expert on this particular topic to digest the current state of play. As someone who is a bit of a generalist in international relations, I find literature reviews extremely helpful, because they let me get up to speed quickly on a new area of research.

More advice here from the Enago blog.

Posted in Uncategorized | Tagged | Leave a comment

The pitfalls of recording interviews

Some sage advice from Telegraph journalist Tom Chivers. For all of those who head out to collect interview data confident that the electronic data recorder in their pocket will meet their every need, always pack a notebook and pen, (better still two pens, even the most basic technology is fallible).


Posted in Uncategorized | Tagged , | Leave a comment

The Conservative Party and immigration: a lesson in bad survey design

Screen Shot 2014-09-23 at 11.50.32For some years as part of our undergraduate research methods teaching we have provided students with a mock-questionnaire which has been deliberately designed to enable them to identify the various potential flaws in survey design, and it is hoped, avoid them when they come to design their own surveys. As is often the case with such exercises the mock-survey has been carefully constructed to include a diverse range of errors and flaws, some obvious and some less so, but far more numerous than one would expect to find in any real survey one is likely to encounter.

Or maybe not. I recently encountered the above survey which is being distributed by the Conservative Party. I wondered at first if it was in fact a parody, someone’s attempt to create a deliberately flawed survey, or an unabashed attempt to make the Conservatives look bad, but it does appear to be genuine.

Leaving aside the politics, the flaws in the survey design are many and obvious, but it may, nevertheless, be instructive to draw some of them out.

  • It is always a good idea to include some kind of introductory section explaining the purpose of your survey, particularly if it is an online survey. However, when doing so one must be careful to avoid doing this in such a way as to reveal to the reader in advance how you would like them to respond to your survey. Similarly, one should avoid associating particular responses with some variable independent of the survey, such as gender (‘men usually respond in this way…’) or political party. In this case the introduction clearly states that Conservatives support a particular position, if you are a Conservative supporter there really is no need to read the rest of the survey one need simply agree with everything that follows.
  • When drafting survey questions be careful to avoid making causal links between unrelated phenomena, particularly if you are not going to provide evidence to support the link. So for example, asking people if they support controlling immigration is OK, but asking if they support it in order to ensure that ‘the economy and local services deliver’ suggests that the two are linked. If you are going to make such links in your questionnaire you really need to be able to back this up with data. It might be an idea to include some independent statistics at this point.
  • Similarly, be sure to collect data on something that matters or is significant. Periodically students will want to carry out research on government planning for alien invasion or zombie attack, and while these are frightening prospects I usually point out that they are also extremely rare or perhaps non-existent phenomena, and they would therefore be better served focusing their research on something more significant. Similarly, while benefits and health tourism may be notionally possible I would want much more information about the scale of the problem before considering it worthwhile to include in a survey such as this. There is also a danger that if you start asking questions about a particular phenomena your research might actually serve to create public concern where it did not previously exist.
  • Try to use clear and unambiguous language and avoid emotive terms or colloquialisms. For example, if you don’t read certain newspapers it may not be entirely clear what is meant by ‘benefits tourism’ or ‘health tourism’, or indeed whether these are good things or not. Similarly, there is a reference to ‘hardworking taxpayers’, who do sound like people I would like to support but I’m not sure that the tax-system, or anything else, discriminates between hardworking and non-hardworking taxpayers, it might be better to refer to ‘taxpayers’ or simply ‘people’. Conversely, I’m not sure what ‘bogus colleges’ are and I’m almost certain I’ve never seen one, but they sound like a bad thing, so how could one object to their closure? As long as its not my son’s college, but I’d never send my child to a bogus college, so closing them should be OK, shouldn’t it?
  • If you are going to use a lot of questions which require a great deal of prior knowledge, particularly if you are not going to provide the respondent with some information to explain your questions, then it is often a good idea to include an additional check-box for ‘don’t know’ responses.
  • Avoid generalising, while the questionnaire offers the reassuring suggestion that it is only bogus colleges which will be shut down, I am a little concerned with the suggestion that immigrants need to speak better English. There seems to be an assumption here that immigrants don’t speak very good English, but a large proportion of immigrants come from English-speaking countries like America, Australia or the Republic of Ireland. It may be better to ask which group of immigrants you would want to speak better English, perhaps breaking it down according to continent, then you could ask whether people think immigrants from Africa, Asia, Europe or America should be able to speak better English. On the other hand, it is possible that people might then misconstrue your meaning and assume you are being racist, so probably best to avoid this one altogether.
  • It is also a good idea to frame your questions in such a way that all of the questions don’t elicit the same response. When completing surveys respondents often get bored and can easily slip into the habit of responding to every question in the same way, repeatedly ticking the yes or no box, particularly if responding to the first few questions in this way corresponds with their actual views. It is better then to change the wording of your questions and frame them in such a way that adopting one particular position does not lead you to respond to all of the questions in the same way. This not only keeps your respondent awake it also means they have to think, and as a result may offer a more considered response. Of course this does mean that they are more likely to spot any flaws in your survey, and may be less likely to give the response you desire.
  • Above all avoid asking questions which are deliberately designed to prompt the respondent to answer in one way or another – these are known as leading questions. This can be done by making causal links between unrelated phenomena, using simplistic or emotive language, basing questions on certain assumptions, and framing questions so that each one prompts the same desired response. This is the cardinal flaw in survey design, it is important to remember at all times that you are conducting research to find out what people think and not to change the way people think, or in order to provide data which will support conclusions which you have already arrived at.

This is then a terrible questionnaire or an excellent example of a poor questionnaire. Any student submitting it as part of an assessment would fail, and it certainly would not be approved by any university research ethics process with which I am familiar. It is almost beyond parody, but then there is this.

Posted in Uncategorized | Tagged , , | Leave a comment