Schoolboy errors in research design, part 1

Reliability is an important concern in research design. It involves ensuring that your research measures what you want it to measure. This may be affected by a number of things including reactivity which is the tendency of research participants to respond in an untypical way because they know they are being studied. This may involve a social desirability or observer effect in which participants, rather than offering an honest response to, for example, survey questions, respond in a way which they believe to be socially acceptable or which they consider is expected by the person carrying out the research.

For example, at the start of a new module a university lecturer gives a small group of eager students a detailed summary of the structure and learning objectives of the module they are about to study. He then asks them to write down what they hope to achieve from the module. The majority of responses bear a remarkably close resemblance to the learning objectives just set out by their tutor. This may mean that the students’ aspirations neatly correspond with the hopes of their tutor. Alternately, it may mean they are conforming to expectations, trying to ingratiate themselves with their tutor, or simply can’t be bothered to think of what they really want and are quite pleased someone has told them what to expect.

Being an experienced researcher I am of course quite happy to dismiss all of this. It is true that at the beginning of this year’s Researching Politics and International Relations module, when students were asked to write down what they hoped to gain from the module, their responses closely mirrored and in some cases exactly matched the learning objectives for the module, but I’m pretty sure this is a clear indication of a well-designed module.

Most students seemed to grasp and appreciate that the module provides important preparation for the third year Independent Study. Many expressed the hope that the module would help them to select a topic for study. They clearly had some idea about what they would like to study and hoped that the module would help them to ‘narrow down’ or ‘further develop’ their ideas. Gratifyingly, a large proportion of respondents hoped that the module would provide them with a deeper understanding of the research process and the necessary ‘research skills’ to enable them to carry out their own research. It is also pleasing to note how many respondents, not only hoped that the module would help them to write their dissertation, but that it will also enhance the quality of their work, leading to ‘good’, ‘effective’ or ‘successful’ work.

A number of respondents hoped that the module would focus in particular on the methods used for research in politics and international relations, while one person hoped the module would be an improvement on the first year research methods module, noting that they hoped the module would cover, ‘the things Applying Research should have taught us.’ Interestingly no less than three individuals referred to their desire to learn more about the statistical package, SPSS, which does feature on the first year module, but not this one. One student, perhaps somewhat sceptical about the modules approach of exposing students to case studies of academic research, observed that, ‘looking at case studies conducted by academics will help in providing in a limited way a model for proper academic research.’ In contrast I was particularly pleased with the respondent who had clearly been listening when I had said that the skills acquired in this module would be applicable throughout their degree programme, and had consequently expressed the hope that the module would ‘increase my abilities in researching which can be useful for all modules.’

There was clearly a nervousness on the part of some respondents about the work involved in producing a substantive piece of independent research. Concerns which the module would certainly hope to address. Several expressed the hope that the module would give them ‘confidence’ to produce their own research, while one rather desperately appealed  ‘what exactly am I supposed to do in 12,500 words.’ However, several individuals seemed to have hopes for the module which, while clearly desirable, went some way beyond the learning objectives and what we might reasonably expect to deliver. One student hoped the module would give them the ‘desire’ to conduct independent research, while another hoped it would instil a ‘passion’ for research. A third with perhaps lower, but no less elusive, expectations hoped the module would ‘encourage me to work hard.’  Given the 9 o’clock scheduling for sessions on this module I am not sure of its ability to meet the hopes of the student who wants the module to provide them with ‘a sensible sleeping pattern’. Similarly, one hopes that the student who would like the module to provide ‘an idea of what I’m doing’ was referring to a proposed research project and not to life more generally.

Finally, a significant number of respondents adopted a more instrumental approach to the module, in that what they hoped for was primarily to pass the module, and/or the Independent Study which follows from it, albeit at various levels. This group was typified by the student who, when asked what they hoped to gain from the module, simply wrote ‘at least a 2:1’, it’s not entirely clear whether this is an aspiration or an instruction.  There is nothing wrong with a goal orientated approach to one’s studies particularly if it means that one is prepared to work hard to reach the desired level. Interestingly, although achieving a particular grade is not one of the learning objectives of the module, there is considerable interest within the government in using student attainment as a performance indicator for university teaching. There are many who would argue that the assumption that degree classifications provide a reliable indicator of teaching quality reflects another schoolboy error in research design, but that is a question for another day.

This entry was posted in Uncategorized. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *