Systematic Reviews are most frequently conducted in the fields of education, health & medicine, social policy or psychology. Increasingly, with the focus on multidisciplinary research, researchers in other areas may find yourselves involved in conducting or supporting a systematic review.
Whilst Durham University Library does not currently offer a dedicated systematic review support service, the Library may be able to help. Whether you are involved in a systematic review for the first time as part of your postgraduate study or a grant proposal, or have previous experience conducting one as part of wider research projects, the resources collected here offer practical guidance and support, and you can contact your department's Liaison Librarian for assistance in constructing effective and comprehensive search strategies and identifying appropriate resources.
"A Systematic Review is a literature review that is designed to locate, appraise and synthesise the best available evidence relating to a specific research question and provide informative and evidence-based answers." (Boland et al, 2014)
It aims to review the available research evidence with the same level of rigour that should be used in producing the research evidence in the first place (Hemingway & Brereton, 2009), taking steps to "reduce/make transparent hidden bias and 'error'" (Newman & Dickson, 2012) and allowing "potentially unmanageable amounts of literature to be managed in a scientifically credibel and reliable way (Torgerson, Hall & Light, 2012).
To achieve this, a Systematic Review seeks to:
- answer a pre-defined and clearly formulated research question.
- use a transparent, comprehensive and pre-defined (to reduce the potential bias in the search and identification process) search strategy to locate the best available evidence, and to allow others to replicate that search process.
- set and apply pre-defined inclusion and exclusion criteria (to reduce potential bias in the selection process) for research evidence identified as part of the search process.
- critically assess the selected studies in terms of quality and relevance, and to identify and account for potential bias.
- provide a balance and impartial summary and interpretation of the findings of the studies, drawing any relevant conclusions or identifying any identified gaps or flaws in the available research evidence.
The end goal is to produce more reliable findings that can be used to inform future decision-making in that field (Higgins & Green, 2011).
Systematic Review vs Literature Review
A Systematic Review is not just a 'big' literature review. Whilst both seek to provide a summary of the available literature on a topic, a Systematic Review is expected to be more rigorous, and ultimately to be transparent and replicable.
|Systematic Review||Literature Review|
|Objectives||Focussed on and by a single, pre-determined research question, with clear objectives identified and stated prior to conducting the primary search of the literature.||Not necessarily focussed on a single research question. May describe an overview around a theme or topic of research. Clear objectives for the literature search may not be clearly identified.|
|Preparation||Methods of identification, selection, evaluation and synthesis are pre-defined to minimise the risk of bias.||Methods of identification, selection, evaluation and synthesis may change subject to authors changing awareness and knowledge of evidence presented, building in the authors subjective knowledge and bias into the review.|
|Design & Methodology||Based on clear identifiable steps with the purpose of being replicable.||No clear rationale, based on expert substantive knowledge of author(s).|
|Selection of Evidence||
Explicit over how reviews included were identified and selected.
|Selection is not made explicit. Unclear whether selected evidence is representative, comprehensive or a (biased) sample.|
|Evaluation of Evidence||All assumptions and judgments are made explicit against identified criteria and open to scrutiny and replication.||Subject to author(s) knowledge and opinion.|
|Outcomes and data synthesis||Clear summary of studies, including an objective assessment of the quality of the evidence presented and potential for bias in the studies selected.||Summary based on studies may not address in detail quality of data or presence of potential bias.|
The above table was based upon comparisons found in Torgerson, Hall and Light (2012) (in Arthur et al. (2012)) and Bettany-Saltikov (2012).
Stages of a Systematic Review
Any Systematic Review should be approached in a methodical way, with several key steps outlined below. There are various ways this process is outlined in the literature, although all roughly following a similar process. Examples form the literature are included in the References section below.
- Define a formulate the research question.
- Define inclusion/exclusion criteria.
- Define a transparent and comprehensive search strategy.
- Search for studies based on pre-defined search strategy.
- Selection of studies based on pre-defined inclusion/exclusion criteria.
- Coding of data collected and selected.
- Critical assessment of studies in terms of quality, relevance, and addressing potential bias in each study.
- Synthesis of findings.
- Interpret the results with an eye on impartiality and reducing potential bias of the authors, and draw any conclusions.
(Boland et al, 2014)
- Definition of question / problem
- Identification of available evidence
- Critical assessment of available evidence
- Synthesis of findings and drawing of conclusion(s)
(Newman & Dickson, 2012)
- Formulate a review question
- Search and selection of studies
- Coding to describe studies
- Critical assessment of studies (quality and relevance)
- Synthesis of results
- Deriving conclusions and implications
(Torgerson, Hall and Light, 2012)
- Transparent, comprehensive search strategy
- Clear pre-specified inclusion/exclusion criteria
- Explicit methods for coding, quality appraising and synthesising included studies
(Cochrane Collaboration, eight stages of doing a systematic review)
- Define review question and develop criteria for inclusion of studies
- Search for studies
- Select studies and collect data
- Assess risk of bias in selected studies
- Analyse data and undertake meta-analyses
- Address reporting biases
- Present results and summary of findings
- Interpret results and draw conclusions
Useful Examples of Systematic Reviews
Below we have tried to provide some useful examples of systematic reviews where you can access and review the search strategy's employed. This may be of interest when approaching your own Systematic Review, but also even if you are not engaged in a systematic review, as visible examples of how other researchers have employed various search tools and techniques to construct a well-focussed but comprehensive search strategy.
- Christie, A. (2014) Are there intervention-generated inequalities in type 2 diabetes care? A systematic review and analysis of routine data. Doctoral thesis, Durham University.
- Doctoral Thesis (Durham): Provides an outline of search strategy and selection of studies (p45-6) and a comprehensive outline of search terms and search strategy employed (Appendix A, p219-221)
- Kyriakou, A. and Higgins, S. (2016) 'Systematic review of the studies examining the impact of the interactive whiteboard on teaching and learning : what we do learn and what we do not.', Preschool and primary education., 4 (2). pp. 254-275.
- Peer-reviewed journal article (Version of Record) (Education): Provides an outline of how the authors approached conducting a systematic review (p258) and the search strategy (p259) and selection and evaluation of studies (p260-269).
- Weed, M. et al (2015) 'The olympic games and raising sport participation: A systematic review of evidence and an interrogation of policy for a demonstration effect.', European Sport Management Quarterly., 15 (2). pp. 195-226.
- Peer-reviewed journal article (Accepted Manuscript) (Applied Social Sciences): Provides an outline of how the authors approached conducting a systematic review (p10-12) and the search strategy (Table 1, p43) and evaluation of studies (Table 2, p44).
- Lockwood,J. et al (2017) 'Impulsivity and self-harm in adolescence: a systematic review', European Child & Adolscent Psychiatry, 26(4). pp. 387-402.
- Peer-reviewed journal article (Version of Record) (Psychology): Provides a summary of methodology employed (p389-90), although an example of where a detailed search strategy is not included within the published article, but is available on request to the corresponding author.
Books in the Library
- Boland, A., Cherry, M. G., & Dickson, R. (2014). Doing a systematic review : a student’s guide
- Bronson, D. E. (2011). Finding and evaluating evidence : systematic reviews and evidence-based practice.
- Newman, N. & Dickson, K. (2012) A Systematic Review in Waring, M., Coe, R., & Arthur, J. (2012). Research methods and methodologies in education.
- Torgerson, C. (2003). Systematic Reviews
- Torgerson, C., Hall, J., & Light, K. (2012) Systematic reviews, in Waring, M., Coe, R., & Arthur, J. (2012). Research methods and methodologies in education.
Online Guides and Resources
- Systematic Reviews: An open Access journal (provided by Biomed Central), encompassing all aspects of the design, conduct and reporting of systematic reviews.
- Systematic Review Toolbox: A community driven resource led by academics at York University, to allow users to search for software tools which can be used to support systematic review activity.
- Cochrane Handbook for Systematic Reviews of Interventions: the official guide that describes in detail the process of preparing and maintaining Cochrane systematic reviews on the effects of healthcare interventions.
- Cochrane Training: An introduction to Systematic Reviews provides an overview of the rationale and process of undertaking a systematic review, with some additional information about the history and aims of Cochrane.
- RGU: Guide to Systematic Reviews
- KCL: Guide to Systematic Reviews
- Wikipedia: Systematic Reviews
Finding Published Systematic Reviews
The resources below are available to identify if a systematic review of your topic of research already exists, beyond subject databases you may already be familiar with where published in academic journal articles.
- Campbell Collaboration Online Library
- Cochrane Database of Systematic Reviews (also including Cochrane Reviews)
- Prospero: International Prospective Register of Systematic Reviews
- PubMed Clinical Studies
Google Scholar and Systematic Reviews
When approaching Systematic Reviews, it is important to be aware of the limitations of using Google Scholar for this purpose.
Coverage and transparency
If a systematic review aims to locate all of the best available evidence relating to a specific research question, then a key aim of the researcher should be to search multiple appropriate sources to ensure a comprehensive search of the available literature.
Google Scholar might seem a logical tool to use, given its breadth of coverage. However, another criteria of a systematic review is transparency in how available research is identified and selected. Here Google Scholar creates problems for the researcher in the context of a Systematic Review:
- There is no content coverage list for Google Scholar: Google Scholar indexes vast amounts of content, from various sources. Unlike many academic databases (e.g. Scopus, Web of Science, ERIC, British Education Index), however, there is no "content coverage list" to clearly indicate what is included (and what is not included) in any search made.
- Content can be added, but is also removed, without any notice: It is possible that a seach conducted as part of a systematic review may identify results one day, which are no longer discoverable at a later date if that content is removed or no longer indexed. It is also not possible, if trying to replicate the search, to identify if a 'new' result identified was just not indexed previously, or was missed by accident or excluded by the researcher for other reasons not specified in the review.
- Not all content is indexed to the same standard: Unlike Google, many publishers will allow web crawlers from Google Scholar to index the full-text of content, to aid discoverability. But not all sources do, which can mean some publications will appear, or be ranked, differently, dependent upon how they are indexed by Google Scholar; academic database would usually indicate differences in coverage and indexing, but Google Scholar offers no transparency for researchers to identify potential factors affecting what results they are returned.
- You can't actually view all of your results: It is not uncommon to find a Google Scholar search returning results in their thousands, ten's of thousands or hundred's of thousands. However, no matter how many results you return, Google Scholar will level let you see or download more than the first 1,001 results.
One advantage Google Scholar is recognised for is its coverage of grey literature, which may not be as well indexed or covered by traditional academic databases, which may focus more on peer-reviewed journal and conference literature.
Google Scholar cannot accurately process any search which uses more than one “concept”, each described by multiple synonyms/alternative terms.
- (teen OR adolescent OR juvenile) AND (achievement OR attainment OR grade)
... will be interpreted by Google Scholar as...
- teen OR adolescent OR juvenile OR achievement OR attainment OR grade
This can have a significant effect on the ability to identify and filter the most appropriate results. This deficiency is exagerrated further where you cannot view any results past the first 1,001 results identified, when they are ranked by Google Scholar's proprietary (and unknown) search and ranking algorithm.
Some studies (Gehanno et al, 2013) highlight that publications found through searching other academic and professional sources can also be found via Google Scholar, and have argued that Google Scholar is therefore a useful source for Systematic Reviews as it's coverage is much broader than other tools. However, other researchers have pointed out that searching for known articles is very different to constructing a comprehensive search protocol using boolean operators to identify unknown research studies.
Search results will be ranked by Google Scholar's proprietary (and unknown) search and ranking algorithm. Whilst unknown, it is apparent that the ranking takes into account various factors, including:
- "Venue" (e.g. journal) of publication
- Citation count
... but also
- If a result has been recently cited
- Search history of the search operator
- Search history of others searching for similar keywords
- Location of the search operator
Given this, it is possible that a search protocol will offer different results, and ranking of results, to different users. This, alongside the problems with the search logic and the lack of a clear (and changing) coverage list, make it difficult to reproduce a search on Google Scholar with the level of certainty a Systematic Review is expected to provide.
- Haddaway NR, Collins AM, Coughlin D, Kirk S (2015) The Role of Google Scholar in Evidence Reviews and Its Applicability to Grey Literature Searching. PLoS ONE 10(9): e0138237. https://doi.org/10.1371/journal.pone.0138237
- Gehanno, J-F., Roolin, L., Darmoni, S. (2013) Is the coverage of google scholar enough to be used alone for systematic reviews. BMC Medical Informatics and Decision Making 13(7) https://doi.org/10.1186/1472-6947-13-7
- Giustini, D. and Boulos, M.N.K. (2013) Google Scholar is not enough to be used alone for systematic reviews. OJPHI 5(2) https://dx.doi.org/10.5210%2Fojphi.v5i2.4623
- Herman, S.W. (2013) Google Scholar Could Be Used as a Stand-Alone Resource for Systematic Reviews. Evidence Based Library and Information Science 13(1) https://doi.org/10.1186/1472-6947-13-7