Critical appraisal has been made much easier by the development of tools that guide you through the process of what aspects of the research study you should be evaluating. Some argue that it isn’t until you become experienced in research methods that you can properly evaluate the quality of the methods used by others. I think this is a valid point but we all have to start somewhere, and critical appraisal is one of the best ways I know to learn about research methods if you aren’t able to actively do a research project.
The quality of a research study is determined by how appropriate the approach, the design and the methods used were to address the question. The first decision we make when evaluating a study therefore is whether this question would be best answered using a qualitative approach (using interviews or focus groups to find out about people’s experiences) or a quantitative approach (collecting data using questionnaires and measurement tools). Some questions are best answered by using a combination of both approaches (mixed methods or multi-methods). Quite often the person undertaking the critical review will decide ahead of time (a priori) which approach is appropriate and use that as an inclusion criterion for the retrieval of papers.
Once you can determine the approach a study has taken you move onto deciding what design the researchers have used. As a novice this can be particularly difficult. If you are lucky the researchers have included the design in the abstract of their paper. If not, you will have to work it out by looking at whether there is a hypothesis or not (usually a sign of a quantitative analytical study) or an aim (usually a sign of an observational study).
Here are some notes about different types of studies, with thanks to Josette Bettany-Saltikov from whose book ‘How to do a systematic literature review in nursing‘ I have adapted the diagrams.
Quantitative designs
Case reports and case series
A report of the treatment of a single patient or more than one patient. Usually published because the condition or treatment is very rare.
Case control studies
Patients who have a condition are compared with those who don’t. The data for the study is taken from the medical records and patient recall (memory).
Cohort studies
This design follows a large group of people for a long time who all have the condition of interest, usually as they receive a treatment, or to observe how the condition changes.
Randomised controlled trials
Used to explore the effect of treatments by comparing a group who are given the treatment with one who are not. The strength of the design is that by randomly allocating patients to the active and control groups you will end up with broadly similar people in each group (in terms of the variables that could inadvertently influence the outcome being measured).
Qualitative designs
Phenomenology
Exploring the lived experience of a group of patients or staff.
Ethnography
Describing a culture – usually by becoming part of that group to understand it. Good to understand how people see their own experiences.
Grounded theory
The researcher aims to develop a theory that explains events and behaviours.
The other aspect to design is to determine when the data was collected and whether it was taken from a source that already existed (e.g. the patients’ notes) or whether it was collected for the study.
- At one point in time = cross-sectional
- At several time points = longitudinal
- Data that already existed and was collected for a different purpose = retrospective
- Data is being collected for the research study = prospective.
Once you have a basic idea of what types of study there can be you can usually work out what the design is even if the authors don’t state it.
You might find this decision tree for research design helpful. Feedback would be very helpful to improve the design of this resource.