top of page
  • Writer's pictureRichard Higgs

Review! Systematic?

The systematic review as a research design is very appealing.


Discovering that it is a research design that does not require you to collect data from participants, and that the ethics review process for systematic reviews is a formality, can induce starry-eyed visions

of completing your dissertation in record time. And if you are an introvert like I am, the idea of minimising your interaction with real living people does sound like a good one.


But things are not quite so simple. The systematic review method is one inherited from Health Sciences, and is typically dependent on highly structured quantitative data. The aim of the method is to evaluate the quality of numerous studies on a particular topic (typically randomised controlled trials - RCTs), pick out those that meet certain quality criteria, and then aggregate the results. The objective is to put together a much bigger sample than is possible in a single trial by combining multiple trials, and thus provide a more reliable conclusion than each of the trials does individually.


This type of meta-analysis has proved very useful of late in determining whether or not the controversial drug Ivermectin is indeed as effective at treating or preventing COVID-19 as some people claim it is. It collects all available relevant RCTs for the drug, eliminates the ones that show obvious bias or faulty methodology, and then mashes the remainder together to provide a very large virtual sample of participants, giving a much larger dataset on which the subsequent statistical analysis can be performed. The results are therefore deemed to be more reliable than the smaller individual studies on their own.


In the LIS discipline, a direct translation of that objective is difficult. We lack the rigorously structured tools and methods of the RCT, and so it is unlikely to be possible to match even the most quantitative data between one study and another. (Some models like LIBQUAL would be an exception, because they always use an identical data collection instrument). Situated as we are in Social Sciences and Humanities, we have to adapt our objectives for using the concept of a meta-analysis, as well as our expectations of the types of research questions that it would be able to answer.


It is arguable whether we could call a review that aggregates and systematically analyses a set of LIS literature a 'systematic review' at all - at least by the formal definition that Health Sciences uses.


This presents the researcher who is attracted by the method with a bit of an ontological bind. Naturally you wish to distinguish from the easygoing structure, inherent bias and lack of comprehensiveness of the literature review that supports your study. You want to use literature as a data source, rather than merely providing context for your research as the standard literature review does, and you want your analysis framework to be able to accommodate the rigour that your would apply to the analysis of any other data source, whether it be the statistical analysis of quantitative data from a questionnaire, or the careful coding of qualitative inputs from interviews or focus groups.


At the same time, it would be inaccurate to make the same claims of providing a larger sample that the meta-analysis of RCTs does in Health Sciences, because you are unlikely to be comparing apples with apples.


Open Government License 2.National Fruit Collection.

That doesn't mean that the concepts and the methods of a systematic review are not useful or applicable for LIS research. But you should be aware of the limitations and applications.


Are there ways around this problem?

The linked article provides a very useful typology that you can draw on, of terms and potential nomenclature that you could draw on to contextualise a consolidated literature study without attracting some of the labyrinthine complications that arise from using the term 'systematic review' without qualifying it very carefully.



14 views0 comments
bottom of page