Schoolboy errors in research design, part 2

In designing a data collection instrument one needs to consider how one is going to record the data. This involves some very basic considerations, for example, in the case of interviews one needs to consider whether one is going to record the interview and transcribe it later or make notes and then type these up. If one is recording a conversation it is important that one can understand everything that is said, sometimes snippets of conversation which seemed perfectly ordinary at the time, can sound muffled or unintelligible on tape.

Similarly if one is making notes then one needs to be able to read one’s own handwriting later – not always an easy task. The question of legibility may be even more problematic if one is working with documents such as handwritten correspondence and papers in an archive, in which one has to decipher someone else’s handwriting. This may also be a problem if you have asked individuals to complete a survey themselves in which they have been asked an open-ended question to which they were required to write a response, such as ‘what do you hope to gain from this module?’  In a recent survey I conducted, whilst most of the responses to this question were legible, I did struggle with one which appeared to say: ‘advice on bags of cheese with pics’. After considerable scrutiny, and not entirely helpful suggestions from others (my wife thought it said ‘advice on logs and cheese’) I concluded that it probably says ‘advice on ways to choose possible topics.’

Posted in Uncategorized | Leave a comment

Schoolboy errors in research design, part 1

Reliability is an important concern in research design. It involves ensuring that your research measures what you want it to measure. This may be affected by a number of things including reactivity which is the tendency of research participants to respond in an untypical way because they know they are being studied. This may involve a social desirability or observer effect in which participants, rather than offering an honest response to, for example, survey questions, respond in a way which they believe to be socially acceptable or which they consider is expected by the person carrying out the research.

For example, at the start of a new module a university lecturer gives a small group of eager students a detailed summary of the structure and learning objectives of the module they are about to study. He then asks them to write down what they hope to achieve from the module. The majority of responses bear a remarkably close resemblance to the learning objectives just set out by their tutor. This may mean that the students’ aspirations neatly correspond with the hopes of their tutor. Alternately, it may mean they are conforming to expectations, trying to ingratiate themselves with their tutor, or simply can’t be bothered to think of what they really want and are quite pleased someone has told them what to expect.

Being an experienced researcher I am of course quite happy to dismiss all of this. It is true that at the beginning of this year’s Researching Politics and International Relations module, when students were asked to write down what they hoped to gain from the module, their responses closely mirrored and in some cases exactly matched the learning objectives for the module, but I’m pretty sure this is a clear indication of a well-designed module.

Most students seemed to grasp and appreciate that the module provides important preparation for the third year Independent Study. Many expressed the hope that the module would help them to select a topic for study. They clearly had some idea about what they would like to study and hoped that the module would help them to ‘narrow down’ or ‘further develop’ their ideas. Gratifyingly, a large proportion of respondents hoped that the module would provide them with a deeper understanding of the research process and the necessary ‘research skills’ to enable them to carry out their own research. It is also pleasing to note how many respondents, not only hoped that the module would help them to write their dissertation, but that it will also enhance the quality of their work, leading to ‘good’, ‘effective’ or ‘successful’ work.

A number of respondents hoped that the module would focus in particular on the methods used for research in politics and international relations, while one person hoped the module would be an improvement on the first year research methods module, noting that they hoped the module would cover, ‘the things Applying Research should have taught us.’ Interestingly no less than three individuals referred to their desire to learn more about the statistical package, SPSS, which does feature on the first year module, but not this one. One student, perhaps somewhat sceptical about the modules approach of exposing students to case studies of academic research, observed that, ‘looking at case studies conducted by academics will help in providing in a limited way a model for proper academic research.’ In contrast I was particularly pleased with the respondent who had clearly been listening when I had said that the skills acquired in this module would be applicable throughout their degree programme, and had consequently expressed the hope that the module would ‘increase my abilities in researching which can be useful for all modules.’

There was clearly a nervousness on the part of some respondents about the work involved in producing a substantive piece of independent research. Concerns which the module would certainly hope to address. Several expressed the hope that the module would give them ‘confidence’ to produce their own research, while one rather desperately appealed  ‘what exactly am I supposed to do in 12,500 words.’ However, several individuals seemed to have hopes for the module which, while clearly desirable, went some way beyond the learning objectives and what we might reasonably expect to deliver. One student hoped the module would give them the ‘desire’ to conduct independent research, while another hoped it would instil a ‘passion’ for research. A third with perhaps lower, but no less elusive, expectations hoped the module would ‘encourage me to work hard.’  Given the 9 o’clock scheduling for sessions on this module I am not sure of its ability to meet the hopes of the student who wants the module to provide them with ‘a sensible sleeping pattern’. Similarly, one hopes that the student who would like the module to provide ‘an idea of what I’m doing’ was referring to a proposed research project and not to life more generally.

Finally, a significant number of respondents adopted a more instrumental approach to the module, in that what they hoped for was primarily to pass the module, and/or the Independent Study which follows from it, albeit at various levels. This group was typified by the student who, when asked what they hoped to gain from the module, simply wrote ‘at least a 2:1’, it’s not entirely clear whether this is an aspiration or an instruction.  There is nothing wrong with a goal orientated approach to one’s studies particularly if it means that one is prepared to work hard to reach the desired level. Interestingly, although achieving a particular grade is not one of the learning objectives of the module, there is considerable interest within the government in using student attainment as a performance indicator for university teaching. There are many who would argue that the assumption that degree classifications provide a reliable indicator of teaching quality reflects another schoolboy error in research design, but that is a question for another day.

Posted in Uncategorized | Leave a comment

Writers on writing: J.K. Rowling

I liked this short, but helpful, advice from J.K. Rowling, who knows a thing or two about writing, in response to a desperate tweet from a student struggling with a dissertation.


Posted in Uncategorized | Tagged , | Leave a comment

The secret to writing

I’ve written before about Oliver Burkeman’s Guardian column, ‘This column will save your life’. In another recent column Burkeman addressed the challenge of dealing with writer’s block. As usual Burkeman draws on a fabled self-help guide, in this case Robert Boice’s, How Writers Journey to Comfort and Fluency. As with all books of fabled repute, Boice’s is long out of print, and changes hands online for large sums. Fortunately, Burkeman shells out £68 for a print-on-demand copy, and rather generously summarises Boice’s advice to save us from doing the same.

He is somewhat disgruntled to discover that Boice’s advice amounts to little more than – write, for a short period, every day. There is a little bit more to it than this, and you should read the column, (and save money on the book), but it boils down to – don’t build writing up to be a big deal, which involves harnessing elusive and mystical creative forces, but rather an everyday activity which you can pick up and put down at will, like running or digging the garden. Only when you have demystified it in this way will you have the freedom to write with comfort and fluency.

Posted in Uncategorized | Tagged | Leave a comment

A note on hypotheses and variables

A hypothesis is a testable statement which seeks to explain a phenomena by examining the relationship between two or more variables. It is important to remember that a hypothesis seeks to explain a phenomena – it does not merely seek to describe it. Statements such as “people don’t like the government”, “people are generally opposed to the legalisation of cannabis” “some people support the death penalty whilst some are opposed” are not hypotheses. They don’t explain a phenomena they simply describe it – there is a lack support for the government, opposition to the legalisation of cannabis, and different views on the death penalty – these things may all be true but they are not explanations, they are not hypotheses.

It is also important to remember that a hypothesis is not a question – “why has support for the government fallen?”, “who supports the death penalty?” “which people are opposed to the legalisation of cannabis?” You may, indeed should, have a research question but a hypothesis posits a potential answer to that question – “Sun readers are more likely to be influenced by the political position of their newspaper than readers of the Daily Mirror”, “women are more supportive of the death penalty than men”, “adults in full-time employment are more likely to be opposed to the legalisation of cannabis than adults in full-time education”.

A hypothesis is a testable statement – it may be true or false – descriptive statements such as “people’s attitudes towards the death penalty”, and questions such as “who supports the death penalty?” are not a testable statements because they can’t be shown to be true or false. Less obviously, the statement “people are opposed to the Government” is not a hypothesis (or at least not a very good one) because it is not really possible to test this statement unless one can prove that nobody is opposed to the government.

In order to show that a hypothesis is not true, one should be able to define a null hypothesis which explains the situation which exists if the hypothesis is proved not to be true. In the case of the hypothesis “women are more in favour of the death penalty than men” the null hypothesis is not that men are more in favour of the death penalty than women (although this would disprove the hypothesis), it is simply sufficient to prove that there is no difference between men’s and women’s attitudes to the death penalty. Establishing a null hypothesis is an effective way of discovering whether you have a statement which can be proved true or not, i.e. a hypothesis.

In order to make their hypotheses testable social scientists introduce the use of variables. These are different factors which may influence the phenomena which is being explained. Hypotheses in the social sciences usually compare two or more variables. So if we use one of the examples offered above, in examining attitudes towards the death penalty one could hypothesise:

“men are likely to be more supportive of the death penalty than women”

In this case gender is being used to explain support for the death penalty, the variables are gender (which may be male or female) and attitudes towards the death penalty (which may be supportive or opposed).

A statement such as “students are in favour of the legalisation of cannabis” – is not a hypothesis. Firstly because it does not seek to explain a phenomena it just describes it. Secondly it doesn’t seek to compare variables. The only variable in this statement is attitudes towards the legalisation of cannabis, “students” is not a variable. There is nothing with which to compare the attitudes of students. One could create a hypothesis on this subject (it’s extraordinary how often students want to look at this issue) along the following lines:

“students are more in favour of the legalisation of cannabis than people who are not students”

However, if this was your hypothesis you would then need to bear in mind that the educational status of your respondents may not be the only factor which explains the phenomena which you have observed. So the factor (or variable) which affects individuals attitude towards the legalisation of cannabis may not be their educational status, but some other variable like their age, gender, religion, or even political affiliation.

The different variables which you identify in testing your hypothesis have different names. The variables for which you are suggesting there is a relationship are called the independent and dependent variables. In this case the independent variable is educational status (whether the respondent is a student or not) this is a fact, or external reality, which will not be changed by the research. The dependent variables are their attitudes towards the legalisation of cannabis – these vary from person to person but are not usually fixed and unchangeable in the way that independent variables like gender, age, social status are. What you are claiming is that people’s attitudes towards cannabis are dependent on their educational status.

This means that such research is essentially predictive. If your hypothesis is proved to be true then knowing someone’s educational status would allow one to predict their attitude towards the legalisation of cannabis.

Any other variables which may have a relationship with the dependent variables but which are not covered by the hypothesis are known as intervening variables – ie they intervene between the suggested relationship between the independent and dependent variable. Intervening variables tend to be other factors similar to the independent variable posited in your hypothesis. So if your hypothesis suggests attitude towards the legalisation of cannabis is dependent on educational status (the independent variable), it may actually be dependent on age, gender or political affiliation – these then are intervening variables.

When designing questionnaires to collect data with which to test a hypothesis you need to try to include questions which establish all the variables you are interested in independent, intervening and dependent. Generally speaking you would begin by covering the independent and any potential intervening variable – Are you a student? What is your gender? which age group do you fit in? What political party did you vote for at the last election? You should then move onto the questions which establish people’s attitudes towards the issue being discussed (the dependent variables), –  Are you in favour of the legalisation of cannabis? Do you think cannabis is less damaging than cigarettes? And so on.

Large scale social attitudes surveys such as British Social Attitudes usually ask a very large number of questions about the background of the individuals being surveyed before moving on to the more substantive questions about attitudes. This enables a large number of independent variables to be tested. Small scale surveys, of the kind carried for most undergraduate research projects are usually more limited, not least because students often find it difficult to persuade a large number of people complete surveys comprised of several hundred questions. This doesn’t mean that such surveys can’t produce valid results but it does mean that careful consideration needs to be given to which variables to include.

Posted in Uncategorized | Leave a comment