Systematic Reviews

Congratulations!

You've decided to conduct a Systematic Review! Please see the associated steps below. You can follow the P-I-E-C-E-S = Plan, Identify, Evaluate, Collect, Explain, Summarize system or any number of systematic review processes available (Foster & Jewell, 2017).

P = Plan: decide on your search methods

Determine your Research Question 

By now you should have identified gaps in the field and have a specific question you are seeking to answer. This will likely have taken several iterations and is the most important part of the Systematic Review process. 

Identify Relevant Systematic Reviews 

Once you've finalized a research question, you should be able to locate existing systematic reviews on or similar to your topic. Existing systematic reviews will be your clues to mine for keywords, sample searches in various databases, and will help your team finalize your review question and develop your inclusion and exclusion criteria. 

Decide on a Protocol and Reporting Standard

Your protocol is essentially a project plan and data management strategy for an objective, reproducible, sound methodology for peer review. The reporting standard or guidelines are NOT a protocols, but rather a set of standards to guide the development of your systematic review. Often they include checklists. It is not required, but highly recommended to follow a reporting standard. 

Protocol registry: Reviewing existing systematic reviews and registering your protocol will increase transparency, minimize bias, and reduce the redundancy of groups working on the same topics (PLoS Medicine Editors, 2011). Protocols can serve as internal or external documents. Protocols can be made public prior to the review. Some registries allow for keeping a protocol private for a set period of time.

Cochrane Database of Systematic Reviews (UGA Login) (Health Sciences)

A collection of regularly updated, systematic reviews of the effects of health care. New reviews are added with each issue of The Cochrane Library Reviews mainly of randomized controlled trials. All reviews have protocols.

PROSPERO (General)

This is an international register of systematic reviews and is public. 

Campbell Corporation (Education & Social Sciences)

Topics covered include Ageing; Business and Management; Climate Solutions; Crime and Justice; Disability;  Education; International Development; Knowledge Translation and Implementation; Methods; Nutrition and Food Security; Sexual Orientation and Gender Identity; Social Welfare; and Training.

Systematic Review for Animals and Food (Vet Med & Animal Science)

Reporting Standards: 

Campbell MECCIR Standards (Education & Social Sciences)

Cochrane Guides & Handbooks (Health & Medical Sciences)

Institute of Medicine of the National Academies: Finding What Works in Healthcare: Standards for Systematic Reviews (healthcare)

PRISMA

• PRISMA for Systematic Review Protocols (General)
• PRISMA Checklist (General)
• PRISMA for Scoping Reviews (General)

Decide on Databases and Grey Literature for Systematic Review Research

Because the purpose of a SR is to find all studies related to your research question, you will need to search multiple databases. You should be able to name the databases you are already familiar with using. Your librarian will be able to recommend additional databases, including some of the following: 

• PubMed (Health Sciences)
• Web of Science 
• Cochrane Database (Biomedical)
• National and Regional Databases (i.e. WHO LILACS scientific health information from Latin America and the Caribbean countries)
• Other discipline specific databases
  • CINAHL (Health Sciences)
  • PsycINFO (Psychology)

Depending on your topic, you may want to search clinical trials and grey literature. See this guide for more on grey literature.

I = Identify: search for studies that match your criteria

Develop Keywords and Write a Search Strategy

Go here for help with writing your search strategy 

Translate Search Strategies

Each database you use will have different methods of searching and resulting search strings, including syntax. Ideally you will create one master keyword list and translate it for each database. Below are tools to assist with translating search strings. 

• Database Syntax Guide (Cochrane)

  Includes syntax for Cochrane Library, EBSCO, ProQuest, Ovid, and POPLINE.

• Systematic Review Search Translator

  The IEBH SR-Accelerator is a suite of tools to assist in speeding up portions of the Systematic Review process, including the Polyglot tool which translates searches across databases. 

• University of Michigan Search 101 - SR Database Cheat Sheet

E = Evaluate: exclude or include studies

Storing, Screening and Full-Text Screening of Your Citations

Because systematic review literature searches may produce thousands of citations and abstracts, the research team will be screening and systematically reviewing large amounts of results. During screening, you will remove duplicates and remove studies that are not relevant to your topic based on a review of titles and abstracts. Of what remains, the full-text screening of the studies will then need to be conducted to confirm that they fit within your inclusion/exclusion criteria.  

The results of the literature review and screening processes are best managed by various tools and software. You can also use a simple form or table to log the relevant information from each study. Consider whether you will be coding your data during the extraction process in your decision on which tool or software to use. Your librarian can consult on which of these is best suited to your research needs.

• EndNote Guide (UGA supported citation tracking software) - great for storing, organizing, and de-duplication
• RefWorks Guide (UGA supported citation tracking software) - great for storing, organizing, and de-duplication
• Rayyan (free service) - great for initial title/abstract screening OR full-text screening as cannot differentiate; not ideal for de-duplication
• Covidence (requires a subscription) - full suite of systematic review tools including meta-analysis
• Combining Software (EndNote, Google Forms, Excel) 
• Forms such as Qualtrics (UGA EITS software) can note who the coder is, creates charts and tables, good when have a review of multiple types of studies

C = Collect: extract and synthesize key data

Data Extraction

Data extraction processes differ between qualitative and quantitative evidence syntheses. In both cases, you must provide the reader with a clear overview of the studies you have included, their similarities and differences, and the findings. Extraction should be done in accordance to pre-established guidelines, such as PRISMA. 

Some systematic reviews contain meta-analysis of the quantitative findings of the results. Consider including a statistician on your team to complete the analysis of all individual study results. Meta-analysis will tell you how much or what the actual results is across the studies and explains results in a measure of variance, typically called a forest plot.

Systematic review price models have changed over the years. Previously, you had to depend on departmental access to software that would cost several hundred dollars. Now that the software is cloud-based, tiered payment systems are now available. Sometimes there is a free tier level, but costs go up for functionality, number or users, or both. Depending on the organization's model, payments may be monthly, annual or per project/review.

TIPS

• Always check your departmental resources before making a purchase.
• View all training videos and other resources before starting your project.
• If your access is limited to a specific amount of time, wait to purchase until the appropriate work stage

Software list

• Abstrackr

  Tool created by Brown University to assist with screening for systematic reviews.

• Cochrane's RevMan

Review Manager (RevMan) is the software used for preparing and maintaining Cochrane Reviews.

• Covidence

Systematic review tool intended to assist with the screening and extraction process. (Requires subscription)

• Distiller SR

DistillerSR is an online application designed specifically for the screening and data extraction phases of a systematic review (Requires subscription) Student and Faculty tiers have monthly pricing with a three month minimum. Number of projects is limited by pricing.

• EPPI Reviewer (requires subscription, free trial)

It includes features such as text mining, data clustering, classification and term extraction

• Rayyan

Rayyan is a free web-based application that can be used to screen titles, abstracts, and full text. Allows for multiple simultaneous users.

• AHRQ's SRDR tool (free) which is web-based and has a training environment, tutorials, and example templates of systematic review data extraction forms
• SUMARI

  "System for the Unified Management, Assessment and Review of Information, the Joanna Briggs Institutes premier software for the systematic review of literature."

• Syras

  Pricing is based on both number of abstracts and number of collaborators. The free tier is limited to 300 abstracts and two collaborators. Rather than monthly pricing, the payment is one-time per project.

E = Explain: give context and rate the strength of the studies

Evidence Synthesis or Critical Appraisal

PRISMA guidelines suggest including critical appraisal of the included studies to assess the risk of bias and to include the assessment in your final manuscripts. There are several appraisal tools available depending on your discipline and area of research.

• Cochrane Training Presentation: Risk of Bias (health sciences)  

  Simple overview of risk of bias assessment, including examples of how to assess and present your conclusions.

• Critical Appraisal Skills Programme (CASP) (various disciplines)

  CASP is an organization that provides resources for healthcare professionals, but their appraisal tools can be used for varying study types across disciplines.

• JBI Critical Appraisal Tools (SRs of primary studies, RCTs, observational studies, and qualitative research) 

  From the Joanna Briggs Institute: "JBI’s critical appraisal tools assist in assessing the trustworthiness, relevance and results of published papers."

• Johns Hopkins Evidence-Based Practice Model (health sciences)

• National Academies of Sciences, Engineering, and Medicine

Document the search; 5.1.6. Include a methods section

List of additional critical appraisal tools from Cardiff University. 

S = Summarize: write and publish your final report

Synthesize, Map, or Describe the Results

Prepare your process and findings in a final manuscript. Be sure to check your PRISMA checklist or other reporting standard. You will want to include the full formatted search strategy for the appendix, as well as include documentation of your search methodology. A convenient way to illustrate this process is through a PRISMA Flow Diagram. 

Attribution: Unless noted otherwise, this section of the guide was adapted from Texas A&M's "Systematic Reviews and Related Evidence Syntheses"