Schoolboy errors in research design, part 1

Reliability is an important concern in research design it involves ensuring that your research measures what you want it to measure. This may be affected by a number of things including reactivity which is the tendency of research participants to respond in an untypical way because they know they are being studied. This may involve a social desirability or observer effect in which participants, rather than offering an honest response to, for example, survey questions, respond in a way which they believe to be socially acceptable or which they consider is expected by the person carrying out the research.

For example, at the start of a new module a university lecturer gives a small group of eager students a detailed summary of the structure and learning objectives of the module they are about to study. He then asks them to write down what they hope to achieve from the module. The majority of responses bear a remarkably close resemblance to the learning objectives just set out by their tutor. This may mean that the students’ aspirations neatly correspond with the hopes of their tutor. Alternately, it may mean they are conforming to expectations, trying to ingratiate themselves with their tutor, or simply can’t be bothered to think of what they really want and are quite pleased someone has told them what to expect.

Being an experienced researcher I am of course quite happy to dismiss all of this. It is true that at the beginning of this year’s Researching Politics and International Relations module, when students were asked to write down what they hoped to gain from the module, their responses closely mirrored and in some cases exactly matched the learning objectives for the module, but I’m pretty sure this is a clear indication of a well-designed module.

Gratifyingly, a large proportion of respondents hope that the module will provide them with a deeper understanding of the research process. Several wrote that they wanted to know ‘how to research properly’, while others hope for an insight into a range of ‘different research methods’. It is also pleasing, particularly given that this is a second year module, that several people referred to their desire for a ‘deeper’ or more ‘in depth’ understanding of research methods. There is also a desire for some specific guidance on how one conducts research specifically in the fields of politics and international relations, which perhaps reflects the fact that the respondents’  experience of research methods teaching up to this point has focused more broadly on social science research methods.

Most students seem to grasp and appreciate that the module provides important preparation for the third year Independent Study. Many expressed the hope that the module would help them to select a topic for study, provide ‘guidance on topic selection’ or help them to find a topic which is both interesting and feasible. Some clearly have some idea about what they would like to study and hope that the module will provide advice on how to ‘further develop’ their ideas. Interestingly, while many respondents were clearly fixated on the monumental task of completing a dissertation by the end of their third year only one respondent hoped that the module would help them to complete a successful research proposal, which is the assessed component of this module.

In addition to providing an insight into the research process and particular research methods, several people also hope that the module will help them more widely to organise their work, and to some extent their lives. There was clearly nervousness on the part of some respondents about the work involved in producing a substantial piece of independent research. Concerns which the module would certainly hope to address. Several expressed the hope that the module would help them learn how to manage a large scale project, manage their own deadlines or become better at time management. However, I am not sure that, on its own, this module will be sufficient to deliver the eight hours sleep which one respondent hopes to achieve.

Finally, a significant number of respondents adopted a more instrumental approach to the module, in that what they hope for is primarily to pass the module, and/or the Independent Study which follows from it, albeit at various levels. Several students said they hope to achieve ‘a good grade’ from the module, others simply want to pass, while the respondent who said that they want ‘not to fail this module’ should perhaps set their sights a little higher. Three respondents were more ambitious stating that what they hope to achieve from the module is a 1st class mark. One, particularly ambitious, or mathematically challenged, individual aspires to achieving ‘at least a 1st’.  Another aspiring 1st class student, should perhaps, learn how to spell dissertation. Interestingly, although achieving a particular grade is not one of the learning objectives of the module, there is considerable interest within the government in using student attainment as a performance indicator for university teaching. There are many who would argue that the assumption that degree classifications provide a reliable indicator of teaching quality reflects another schoolboy error in research design.

This entry was posted in Uncategorized and tagged , . Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>