Book summary “Doing a Systematic Review”: summary

Introduction

You often face an overwhelming volume of published research when you try to answer a specific clinical or scientific question. Traditional narrative literature reviews frequently suffer from selection bias, lack transparent search methods, and yield subjective conclusions that confuse decision-making.

A systematic review is a structured, transparent literature review designed to locate, appraise, and synthesize the best available evidence to answer a specific research question.

This book provides a comprehensive ten-step guide to conducting your review from start to finish. You will learn how to plan the project, design search strategies, extract data, assess study quality, and disseminate your findings.

As you read, you will find answers to several core questions:

• How do you formulate a precise research question?
• What methods ensure an exhaustive literature search?
• How do you critically appraise included studies?
• When should you perform a meta-analysis?

1. Introduction to Systematic Reviews

A systematic review functions as the gold standard for research synthesis by replacing subjective narrative summaries with transparent and reproducible methods.

What is a systematic review

A systematic review is a structured literature review designed to locate, appraise, and synthesize the best available evidence to answer a specific research question transparently.

You must follow well-defined, reproducible steps requiring a clear research question, critical appraisal of evidence, synthesis of findings, and evidence-based conclusions. Different literature review types vary in breadth and depth, with systematic reviews prioritizing depth over broad superficial coverage.

Narrative reviews cover great breadth with little depth. Rapid reviews take shortcuts due to time constraints. Scoping reviews map out evidence breadth without deep analysis. Integrative reviews combine qualitative and quantitative evidence. Realist reviews use existing theory to explain context and mechanisms. Evidence maps present broad visual or database summaries.

Conducting a systematic review for a postgraduate thesis offers strong control over learning objectives and avoids participant recruitment, but is time-consuming and isolating. You gain critical appraisal skills and insight into published literature without needing formal ethical approval or primary research recruitment, though you remain dependent on available published information quality.

Evolution of the process

Systematic reviewing evolved from early efforts to evaluate medical treatments and combine statistical results from independent studies.

James Lind brought together data in 1753 on scurvy prevention in sailors to remove a great deal of rubbish and exhibit a full and impartial view of published literature. Archie Cochrane proposed proper evaluation of healthcare in 1972 and lamented the lack of organized critical summaries of randomized controlled trials in 1979. Gene Glass led the development of statistical procedures for combining study results in 1976, coining the term meta-analysis. The international Cochrane Collaboration network was established in 1992 and now includes 37000 dedicated people across 130 countries. The Campbell Collaboration was established in 2000.

Conducting a systematic review involves ten distinct, structured steps that guide you from initial planning to final dissemination:

• Step 1: Planning your review
• Step 2: Performing scoping searches, identifying the review question and writing your protocol
• Step 3: Literature searching
• Step 4: Screening titles and abstracts
• Step 5: Obtaining papers
• Step 6: Selecting full-text papers
• Step 7: Data extraction
• Step 8: Quality assessment
• Step 9: Analysis and synthesis
• Step 10: Writing up, editing and disseminating

2. Planning and Managing Your Review

Rigorous planning, backwards scheduling, and secure electronic file organization prevent data loss and keep your project on track.

Review processes tend to expand to fill available time. You must identify the thesis submission deadline and work backwards from it to establish realistic goals. Overestimate rather than underestimate the time required for key tasks.

You should write a review protocol at the outset to establish your methods and guide you when feeling overwhelmed. A review protocol summarizes your planned methods and acts as a route map. It is typically read only by your supervisor and student unless registered online.

To put these principles into action:

• Identify the submission deadline for your thesis
• Plan backwards from the submission date
• Overestimate the time required to complete key tasks
• Write a review protocol detailing proposed methods
• Check website resources for published protocol examples
• Consult chapter checklists to guide protocol development

Managing time and resources

You must set up logical folders and subfolders for information storage to maintain order. Include dates in file names for substantive updates to ensure reliable version control. Move older file versions into an archive folder. Back up your work regularly to your computer, email, external drives, or institutional storage. Assign unique study labels consistently.

Establishing consistency across terminology, formatting, abbreviations, and table styling from day one prevents time-consuming edits later. You must choose preferred terms for interchangeable concepts and stick with them. Use abbreviations consistently throughout the text. Maintain uniform capitalization and hyphenation. List included studies in tables alphabetically or chronologically in the same way.

Developing the review question

Developing a systematic review question requires a structured six-step sequence from topic selection to protocol writing. Early scoping searches gauge the volume and type of available evidence and check whether a similar review already exists. You can conduct preliminary literature searches across bibliographic databases and Google Scholar. Search PROSPERO to check for ongoing reviews and examine published reviews for overlap and methodological quality.

Inclusion and exclusion criteria define the specific attributes required for a study to enter the review. These criteria are often structured via the PICOSS framework, which stands for population, intervention, comparator, outcome, study design, and setting. Clear criteria prevent bias and ensure the evidence matches your research question. Contacting topic experts and considering patient and public involvement ensures your review questions remain novel, relevant, and clinically grounded.

You follow these specific steps:

• Step 1: Identify a topic area of interest to you
• Step 2: Carry out early scoping searches
• Step 3: Focus your ideas to define the scope of the review
• Step 4: Finalize your review question and develop your inclusion and exclusion criteria
• Step 5: Consider contacting experts in the topic area
• Step 6: Write a review protocol
• Generate a PICO or PICOSS table for intervention effectiveness reviews
• Define clear inclusion and exclusion criteria mapping directly onto the review question
• Account for language bias, publication bias, and format bias
• Choose preferred terms for interchangeable concepts and stick with them
• Use abbreviations consistently throughout the text
• Maintain uniform capitalization and hyphenation
• List included studies in tables alphabetically or chronologically in the same way
• Draft a specific, targeted email to published experts outlining your review question
• Ask experts about missed or upcoming reviews in the next 3 months
• Incorporate patient and public involvement to align outcome measures with stakeholder perspectives

3. Searching and Selecting Studies

Comprehensive literature searches using advanced database syntax and explicit screening tools ensure all relevant evidence is captured without bias.

Developing search strategies

Scoping searches are initial broad searches using simple search terms designed to provide an overview of the literature and refine the review question. Unlike main searches, scoping searches do not require searching every available resource and help determine the volume of studies likely to be found, thereby assisting in protocol development.

Main bibliographic searches must balance sensitivity and specificity to capture relevant evidence without being overwhelmed by irrelevant citations. Because you face time and resource constraints, a pragmatic approach is recommended rather than attempting an exhaustive search for every obscure piece of evidence.

Systematic reviews should consider both published literature from commercial publishers and grey literature controlled outside commercial channels. Grey literature includes legislation, government documents, annual reports, theses, conference proceedings, and research registers, which prevent publication bias for certain review topics.

Bibliographic database searches utilize advanced syntax including Boolean operators, free-text words, wildcards, and subject headings to retrieve precise records. Databases use operators like AND, OR, and NOT, wildcards like question marks or asterisks for spelling variations, and structured terms like Medical Subject Headings.

Complementary searching activities such as hand searching, citation chaining, and contacting authors are necessary to uncover studies missed by electronic databases. Citation chaining involves backward searching through bibliographies and forward searching through papers citing a key reference, while contacting authors identifies unpublished data or works in progress.

Cherry et al. (2017) conducted a systematic review examining carers of people with mental health difficulties, searching five bibliographic databases followed by checking conference proceedings, reference lists, and contacting corresponding authors. These searches covered inception until October 2015 and were repeated in October 2016.

Screening and selection

The screening and selection of studies from search results is conducted in two distinct stages to ensure eligibility against inclusion and exclusion criteria. Stage 1 involves screening titles and abstracts of all unique citations, while Stage 2 involves evaluating full-text papers using a piloted screening tool and recording exclusion reasons.

Electronic and hand searches identified 2,228 citations in a real study case example, resulting in 2,000 unique citations after de-duplication, 20 potential citations retained after Stage 1 screening, 16 full-text papers evaluated at Stage 2, 12 citations excluded at Stage 2, and 1 additional citation added via an expert, totaling 5 total citations included in the final review.

• De-duplicate references
• Develop and pilot your screening and selection tool
• Screen all titles and abstracts identified via searches against your inclusion and exclusion criteria (Stage 1)
• Obtain the full-text papers of all potentially eligible references
• Use your screening and selection tool to help you identify full-text papers for inclusion in the review (Stage 2)

4. Data Extraction and Quality Assessment

Extracting descriptive data into standardized tables and critically appraising study designs protect your review from hidden biases and flaws.

Data extraction officially begins after applying inclusion and exclusion criteria. You can draft your data extraction form in parallel with writing your protocol. Pilot your form on two studies if your searches identified four or five studies. Pilot your form on four or five studies if you have 15 included studies.

• Design a draft of your data extraction form early, potentially in parallel with writing your protocol
• Pilot your data extraction form on two studies if your searches identified four or five studies
• Pilot your form on four or five studies if you have 15 included studies
• Record numbers and percentages on your form even if you do not describe both in the write-up
• Treat multiple publications of the same study as one paper by extracting all data onto the same form

Data extraction practicalities

Narrative synthesis uses words and references to data tables to present results without repeating every individual tabular entry. You should describe overall findings rather than waste words repeating individual data points already visible in tables. Tabular data should present descriptive characteristics and specific study results. Broader interpretations must wait for the discussion section.

Five hypothetical studies were carried out from 1990 to 2002. These studies had a publication date range from 1993 to 2005.

• Present descriptive data summaries for included studies in a series of tables
• Split descriptive data across a Study Characteristics table and a Participant Characteristics table if necessary
• Accompany all data tables with explanatory summary text that checks for text-table correspondence
• Highlight any key differences in values reported by different studies
• Include an explanation in your thesis that justifies your decision if data are not appropriate for meta-analysis

Quality assessment fundamentals

Publication in a peer-reviewed journal does not guarantee good quality. You must examine flaws, weaknesses, and potential sources of bias such as selection, allocation, detection, attrition, and reporting bias.

In a review of chemotherapy treatments for non-small cell lung cancer by Brown et al. (2013), all 23 included studies were published randomized controlled trials in highly regarded journals, yet the overall quality was concluded to be poor. A separate review of biofeedback for hypertension by Greenhalgh et al. (2009) included 36 randomized controlled trials.

• Note the design of your included studies
• Identify the type of quality assessment tool to suit your review
• Choose the appropriate quality assessment tool
• Carry out quality assessment using the appropriate tool
• Tabulate and summarize the results of your quality assessment
• Think about how the results of your quality assessment impact on the conclusions and recommendations of your systematic review
• Treat the paper with key study data as the core paper during quality assessment when multiple papers exist
• Add footnotes to your quality assessment table to explain the source of information derived from supplements or abstracts
• Conduct a sensitivity analysis excluding poor-quality studies if planning a meta-analysis

5. Analysis, Synthesis, and Dissemination

Synthesizing numerical and qualitative data through meta-analysis or narrative reporting leads to robust conclusions that must be actively disseminated.

Time-to-event outcomes measure the duration until a participant experiences an event, utilizing survival analysis and hazard ratios. Participants who do not experience the event during the observation period are classed as censored. Survival analysis quantifies data into hazards, and hazard ratios are interpreted similarly to relative risks, reflecting the risk of experiencing an event in one group compared to another over a given time period. An example is a hazard ratio of 0.43, meaning the risk of experiencing an event in one group is 43 percent of the risk in another group.

A meta-analysis requires satisfying the assumption of homogeneity across randomized controlled trials before pooling data. Before combining studies, researchers must assess four aspects of homogeneity: similarity of recruited patients via inclusion criteria and baseline characteristics, consistency of interventions and comparators, reporting of the same outcomes over comparable time frames, and description of similar effects with overlapping confidence intervals.

• Step 1: Assess whether it is appropriate to combine your studies in a meta-analysis by checking the assumption of homogeneity
• Step 2: Justify your decision to conduct or not conduct a meta-analysis in your text
• Step 3: Choose an appropriate meta-analysis method
• Step 4: Identify and discuss any heterogeneity in the meta-analysis results
• Step 5: Present and interpret the results of your meta-analysis

Meta-analysis and heterogeneity

Meta-analysis calculations must preserve individual study randomization and apply a weighted average where larger studies take precedence. Lumping participants together or calculating a simple arithmetic mean introduces bias because small studies are less accurate and are assigned less weight. Meta-analyses use fixed effects models, assuming a single true effect with variability due to chance, or random effects models, assuming true effects vary across studies around an average.

Heterogeneity measures the variability between studies and is categorized into clinical, methodological, and statistical types. Reviewers can visually inspect forest plots for overlapping confidence intervals, apply a chi-squared test for heterogeneity using a significance threshold of $p$ equals 0.1, and calculate the $I^2$ statistic to quantify the percentage of total variation due to heterogeneity rather than chance. Numerical values include 0 percent indicating no observed heterogeneity, 25 percent representing low heterogeneity, 50 percent representing moderate heterogeneity, and 75 percent representing high heterogeneity.

The discussion and conclusions sections must provide a critical interpretation of review results in relation to the review question rather than repeating individual studies. Examiners heavily scrutinize the discussion and conclusions. Students should structure these sections by reviewing the question and methodology, summarizing main findings, fitting findings into published research, examining strengths and limitations, assessing generalizability, and detailing implications and final conclusions.

• Start with a brief overview of the review question and methodology
• State the main findings of the review
• Explain how findings fit with previously published research
• Analyze the strengths and limitations of the included studies
• Analyze the strengths and limitations of the review process
• Determine if findings can be generalized
• Discuss the implications of the review
• Draw final conclusions from the review

Dissemination and publication

Disseminating systematic review results through traditional and alternative methods maximizes research impact beyond an academic exercise. Reviewers should act promptly after thesis submission, identify target audiences such as researchers, clinicians, policymakers, and the public, and utilize creative channels ranging from peer-reviewed journals and conference proceedings to social media and newsletters.

Publishing a systematic review in a peer-reviewed academic journal requires following an eight-step procedure, from checking suitability and selecting an appropriate journal to responding to reviewers. Journals have stringent standards, and publishing can be time-consuming. Authors must evaluate scientific rigor, timeliness, journal impact factors, open-access costs, word limits, and authorship order before submitting.

• Think very carefully about the suitability of your review for publication in a peer-reviewed academic journal
• Pick an appropriate journal
• Decide on authorship (and author order)
• Turn your thesis into a paper
• Write your covering letter
• Submit your paper
• Receive/respond to reviewers’/editors’ comments
• Receive final outcome

10 Key Ideas

1. A systematic review is a structured literature review designed to locate, appraise, and synthesize the best available evidence to answer a specific research question transparently.
2. Rigorous planning, backwards scheduling, and secure electronic file organization prevent data loss and keep your project on track.
3. Comprehensive literature searches using advanced database syntax and explicit screening tools ensure all relevant evidence is captured without bias.
4. The screening and selection of studies from search results is conducted in two distinct stages to ensure eligibility against inclusion and exclusion criteria.
5. Extracting descriptive data into standardized tables and critically appraising study designs protect your review from hidden biases and flaws.
6. Publication in a peer-reviewed journal does not guarantee good quality because you must still examine flaws, weaknesses, and potential sources of bias such as selection, allocation, detection, attrition, and reporting bias.
7. A meta-analysis requires satisfying the assumption of homogeneity across randomized controlled trials before pooling data.
8. Heterogeneity measures the variability between studies and is categorized into clinical, methodological, and statistical types.
9. The discussion and conclusions sections must provide a critical interpretation of review results in relation to the review question rather than repeating individual studies.
10. Disseminating systematic review results through traditional and alternative methods maximizes research impact beyond an academic exercise.