I liked this short, but helpful, advice from J.K. Rowling, who knows a thing or two about writing, in response to a desperate tweet from a student struggling with a dissertation.
In designing a data collection instrument one needs to consider how one is going to record the data. This involves some very basic considerations, for example, in the case of interviews one needs to consider whether one is going to record the interview and transcribe it later or make notes and then type these up. If one is recording a conversation it is important that one can understand everything that is said, sometimes snippets of conversation which seemed perfectly ordinary at the time, can sound muffled or unintelligible on tape. Similarly if one is making notes then one needs to be able to read one’s own handwriting later – not always an easy task.
The question of legibility may be even more problematic if one is working with documents such as handwritten correspondence and papers in an archive, in which one has to decipher someone else’s handwriting. This may also be a problem if you have asked individuals to complete a survey themselves in which they have been asked an open-ended question to which they were required to write a response, such as ‘what do you hope to gain from this module?’ In a recent survey I conducted, whilst most of the responses to this question were legible, I did struggle with one handwritten response which appeared to say: ‘advice on bags of cheese with pics’. After considerable scrutiny, and not entirely helpful suggestions from others (my wife, a primary school teacher and therefore accustomed to deciphering handwriting, thought it said ‘advice on logs and cheese’), I finally concluded that it probably says ‘advice on ways to choose possible topics.’
Reliability is an important concern in research design it involves ensuring that your research measures what you want it to measure. This may be affected by a number of things including reactivity which is the tendency of research participants to respond in an untypical way because they know they are being studied. This may involve a social desirability or observer effect in which participants, rather than offering an honest response to, for example, survey questions, respond in a way which they believe to be socially acceptable or which they consider is expected by the person carrying out the research.
For example, at the start of a new module a university lecturer gives a small group of eager students a detailed summary of the structure and learning objectives of the module they are about to study. He then asks them to write down what they hope to achieve from the module. The majority of responses bear a remarkably close resemblance to the learning objectives just set out by their tutor. This may mean that the students’ aspirations neatly correspond with the hopes of their tutor. Alternately, it may mean they are conforming to expectations, trying to ingratiate themselves with their tutor, or simply can’t be bothered to think of what they really want and are quite pleased someone has told them what to expect.
Being an experienced researcher I am of course quite happy to dismiss all of this. It is true that at the beginning of this year’s Researching Politics and International Relations module, when students were asked to write down what they hoped to gain from the module, their responses closely mirrored and in some cases exactly matched the learning objectives for the module, but I’m pretty sure this is a clear indication of a well-designed module.
Gratifyingly, a large proportion of respondents hope that the module will provide them with a deeper understanding of the research process. Several wrote that they wanted to know ‘how to research properly’, while others hope for an insight into a range of ‘different research methods’. It is also pleasing, particularly given that this is a second year module, that several people referred to their desire for a ‘deeper’ or more ‘in depth’ understanding of research methods. There is also a desire for some specific guidance on how one conducts research specifically in the fields of politics and international relations, which perhaps reflects the fact that the respondents’ experience of research methods teaching up to this point has focused more broadly on social science research methods.
Most students seem to grasp and appreciate that the module provides important preparation for the third year Independent Study. Many expressed the hope that the module would help them to select a topic for study, provide ‘guidance on topic selection’ or help them to find a topic which is both interesting and feasible. Some clearly have some idea about what they would like to study and hope that the module will provide advice on how to ‘further develop’ their ideas. Interestingly, while many respondents were clearly fixated on the monumental task of completing a dissertation by the end of their third year only one respondent hoped that the module would help them to complete a successful research proposal, which is the assessed component of this module.
In addition to providing an insight into the research process and particular research methods, several people also hope that the module will help them more widely to organise their work, and to some extent their lives. There was clearly nervousness on the part of some respondents about the work involved in producing a substantial piece of independent research. Concerns which the module would certainly hope to address. Several expressed the hope that the module would help them learn how to manage a large scale project, manage their own deadlines or become better at time management. However, I am not sure that, on its own, this module will be sufficient to deliver the eight hours sleep which one respondent hopes to achieve.
Finally, a significant number of respondents adopted a more instrumental approach to the module, in that what they hope for is primarily to pass the module, and/or the Independent Study which follows from it, albeit at various levels. Several students said they hope to achieve ‘a good grade’ from the module, others simply want to pass, while the respondent who said that they want ‘not to fail this module’ should perhaps set their sights a little higher. Three respondents were more ambitious stating that what they hope to achieve from the module is a 1st class mark. One, particularly ambitious, or mathematically challenged, individual aspires to achieving ‘at least a 1st’. Another aspiring 1st class student, should perhaps, learn how to spell dissertation. Interestingly, although achieving a particular grade is not one of the learning objectives of the module, there is considerable interest within the government in using student attainment as a performance indicator for university teaching. There are many who would argue that the assumption that degree classifications provide a reliable indicator of teaching quality reflects another schoolboy error in research design.
I’ve written before about Oliver Burkeman’s Guardian column, ‘This column will save your life’. In another recent column Burkeman addressed the challenge of dealing with writer’s block. As usual Burkeman draws on a fabled self-help guide, in this case Robert Boice’s, How Writers Journey to Comfort and Fluency. As with all books of fabled repute, Boice’s is long out of print, and changes hands online for large sums. Fortunately, Burkeman shells out £68 for a print-on-demand copy, and rather generously summarises Boice’s advice to save us from doing the same.
He is somewhat disgruntled to discover that Boice’s advice amounts to little more than – write, for a short period, every day. There is a little bit more to it than this, and you should read the column, (and save money on the book), but it boils down to – don’t build writing up to be a big deal, which involves harnessing elusive and mystical creative forces, but rather an everyday activity which you can pick up and put down at will, like running or digging the garden. Only when you have demystified it in this way will you have the freedom to write with comfort and fluency.
A hypothesis is a testable statement which seeks to explain a phenomena by examining the relationship between two or more variables. It is important to remember that a hypothesis seeks to explain a phenomena – it does not merely seek to describe it. Statements such as “people don’t like the government”, “people are generally opposed to the legalisation of cannabis” “some people support the death penalty whilst some are opposed” are not hypotheses. They don’t explain a phenomena they simply describe it – there is a lack support for the government, opposition to the legalisation of cannabis, and different views on the death penalty – these things may all be true but they are not explanations, they are not hypotheses.
It is also important to remember that a hypothesis is not a question – “why has support for the government fallen?”, “who supports the death penalty?” “which people are opposed to the legalisation of cannabis?” You may, indeed should, have a research question but a hypothesis posits a potential answer to that question – “Sun readers are more likely to be influenced by the political position of their newspaper than readers of the Daily Mirror”, “women are more supportive of the death penalty than men”, “adults in full-time employment are more likely to be opposed to the legalisation of cannabis than adults in full-time education”.
A hypothesis is a testable statement – it may be true or false – descriptive statements such as “people’s attitudes towards the death penalty”, and questions such as “who supports the death penalty?” are not a testable statements because they can’t be shown to be true or false. Less obviously, the statement “people are opposed to the Government” is not a hypothesis (or at least not a very good one) because it is not really possible to test this statement unless one can prove that nobody is opposed to the government.
In order to show that a hypothesis is not true, one should be able to define a null hypothesis which explains the situation which exists if the hypothesis is proved not to be true. In the case of the hypothesis “women are more in favour of the death penalty than men” the null hypothesis is not that men are more in favour of the death penalty than women (although this would disprove the hypothesis), it is simply sufficient to prove that there is no difference between men’s and women’s attitudes to the death penalty. Establishing a null hypothesis is an effective way of discovering whether you have a statement which can be proved true or not, i.e. a hypothesis.
In order to make their hypotheses testable social scientists introduce the use of variables. These are different factors which may influence the phenomena which is being explained. Hypotheses in the social sciences usually compare two or more variables. So if we use one of the examples offered above, in examining attitudes towards the death penalty one could hypothesise:
“men are likely to be more supportive of the death penalty than women”
In this case gender is being used to explain support for the death penalty, the variables are gender (which may be male or female) and attitudes towards the death penalty (which may be supportive or opposed).
A statement such as “students are in favour of the legalisation of cannabis” – is not a hypothesis. Firstly because it does not seek to explain a phenomena it just describes it. Secondly it doesn’t seek to compare variables. The only variable in this statement is attitudes towards the legalisation of cannabis, “students” is not a variable. There is nothing with which to compare the attitudes of students. One could create a hypothesis on this subject (it’s extraordinary how often students want to look at this issue) along the following lines:
“students are more in favour of the legalisation of cannabis than people who are not students”
However, if this was your hypothesis you would then need to bear in mind that the educational status of your respondents may not be the only factor which explains the phenomena which you have observed. So the factor (or variable) which affects individuals attitude towards the legalisation of cannabis may not be their educational status, but some other variable like their age, gender, religion, or even political affiliation.
The different variables which you identify in testing your hypothesis have different names. The variables for which you are suggesting there is a relationship are called the independent and dependent variables. In this case the independent variable is educational status (whether the respondent is a student or not) this is a fact, or external reality, which will not be changed by the research. The dependent variables are their attitudes towards the legalisation of cannabis – these vary from person to person but are not usually fixed and unchangeable in the way that independent variables like gender, age, social status are. What you are claiming is that people’s attitudes towards cannabis are dependent on their educational status.
This means that such research is essentially predictive. If your hypothesis is proved to be true then knowing someone’s educational status would allow one to predict their attitude towards the legalisation of cannabis.
Any other variables which may have a relationship with the dependent variables but which are not covered by the hypothesis are known as intervening variables – ie they intervene between the suggested relationship between the independent and dependent variable. Intervening variables tend to be other factors similar to the independent variable posited in your hypothesis. So if your hypothesis suggests attitude towards the legalisation of cannabis is dependent on educational status (the independent variable), it may actually be dependent on age, gender or political affiliation – these then are intervening variables.
When designing questionnaires to collect data with which to test a hypothesis you need to try to include questions which establish all the variables you are interested in independent, intervening and dependent. Generally speaking you would begin by covering the independent and any potential intervening variable – Are you a student? What is your gender? which age group do you fit in? What political party did you vote for at the last election? You should then move onto the questions which establish people’s attitudes towards the issue being discussed (the dependent variables), – Are you in favour of the legalisation of cannabis? Do you think cannabis is less damaging than cigarettes? And so on.
Large scale social attitudes surveys such as British Social Attitudes usually ask a very large number of questions about the background of the individuals being surveyed before moving on to the more substantive questions about attitudes. This enables a large number of independent variables to be tested. Small scale surveys, of the kind carried for most undergraduate research projects are usually more limited, not least because students often find it difficult to persuade a large number of people complete surveys comprised of several hundred questions. This doesn’t mean that such surveys can’t produce valid results but it does mean that careful consideration needs to be given to which variables to include.