How to conduct a systematic review, step by step
Chapter 4: Searching the Literature — Video 4
https://ghrbook.com/videos/conduct-a-systematic-review/

[Slide 1] This video walks through how a systematic review actually gets conducted, from the team you assemble and the question you frame, through the search, the screening, and the appraisal, to the flow diagram you publish so a reader can check your work.
[Slide 2] When do you need to be systematic? When your goal is to find all the relevant evidence on a question. Not just enough to orient yourself or support an argument, but everything that exists. This is the foundation of a systematic review: a comprehensive, unbiased synthesis of what the research actually shows.
[Slide 3] Systematic reviews answer questions like: does this intervention work, and to what extent? What does the totality of evidence suggest? Where are the gaps? To answer them fairly, you can't cherry-pick convenient studies. You need to surface all the knowledge, including studies that are harder to find, written in other languages, published in obscure journals, or sitting in grey literature repositories.
[Slide 4] This matters because incomplete searching biases your conclusions. The studies you miss are often systematically different from the ones you find easily. They may have null or negative results, come from underrepresented regions, or challenge dominant narratives.
[Slide 5] The good news is that systematic searching is a learnable skill with clear steps. The bad news is that it takes time, and there are few shortcuts that don't compromise quality. Buckle up, because the systematic review timeline is measured in months and years, not days and weeks.
[Slide 6] Step zero. Systematic reviews are a team sport. They are a lot of work to complete, and you need a variety of voices to be a check on the process. If you're Danny Ocean putting together your crack team, your first pick-up should probably be a librarian. Research librarians are trained in systematic search methods and know database quirks you've never heard of. Many universities offer librarian consultations specifically for systematic reviews.
[Slide 7] You'll also need content experts who understand the conditions, interventions, and exposures. Methodological experts who can help with critical appraisal. And eager teammates willing to screen records and extract data. If your systematic review will include a quantitative technique called a meta-analysis, you'll need someone to lead this work.
[Slide 8] Step one. Your starting point is a well-formulated question. It should be specific enough to search but broad enough to capture relevant evidence. Too broad: how can we improve health in Africa? Too narrow: what is the effect of the ASHA program on infant vaccination rates in rural Uttar Pradesh between 2015 and 2018?
[Slide 9] Just right: what is the effectiveness of community health worker programs for improving childhood vaccination coverage in low-income countries? A good strategy for developing a well-formulated question is to use a framework like PICO. Population: who are you studying? Intervention or exposure: what are you interested in? Comparison: what is the comparison, if any? And outcome: what outcomes matter?
[Slide 10] Step two. With your question formulated and your team assembled, it's time to build the search strategy that will actually find the evidence. A search strategy has two main components: the search terms, meaning the words you'll search for, and the database queries, meaning the structured searches you'll run in specific databases. Start with the terms. Break your question into concepts and identify all the ways those concepts might be expressed in the literature.
[Slide 11] For each concept, generate primary terms, the obvious words researchers use. Synonyms. Related terms, broader or narrower concepts that might capture relevant literature. British and American spellings. Abbreviations and acronyms. And historical terms that might appear in older literature. For community health workers, your list might include community health worker, village health worker, lay health worker, health extension worker, barefoot doctor, promotora, and ASHA in India. Do this for each concept in your question.
[Slide 12] Major databases also use controlled vocabulary: standardized terms assigned to articles by indexers. In PubMed these are called MeSH, or Medical Subject Headings. In Embase they're called Emtree terms. CINAHL has its own headings. Controlled vocabulary thesauruses are helpful because there are many ways to refer to the same phenomenon. The MeSH term for breast cancer is breast neoplasm.
[Slide 13] A search for breast neoplasm in PubMed actually searches more than 30 entry terms. So a sophisticated search uses both keyword searching, to catch new articles not yet indexed, and controlled vocabulary, to comprehensively capture the indexed literature.
[Slide 14] Once you have your terms, you need to combine them into a search query using Boolean operators. OR expands your search: it finds articles with any of the terms. AND narrows your search: it finds articles with all of the terms. NOT excludes terms, and you should use it cautiously, because you might exclude relevant articles. Use parentheses to group terms and control how the Boolean operators apply.
[Slide 15] A short aside on where that word comes from. Boolean refers to a data type with only two possible values: true or false. The term honors George Boole, a nineteenth-century English mathematician who developed an algebraic system for logical reasoning. When you search malaria AND pregnancy, you're asking: is malaria present, true or false, AND is pregnancy present? Only records where both are true get returned.
[Slide 16] Database searching alone won't find everything. For comprehensive systematic reviews, you'll also need to search trial registries, which list registered trials, many of which are never published. Check reference lists: backward citation searching finds older relevant studies, while forward citation searching finds newer ones. Search grey literature, including UN agency repositories, ministry of health websites, and organizational reports. These often contain evidence that never appears in indexed journals, which is particularly important in global health, where much implementation evidence lives outside academic publishing. And contact experts directly, asking about unpublished work, ongoing studies, or reports you might have missed.
[Slide 17] Step three. Before searching, specify exactly which studies you'll include and exclude. Clear criteria prevent bias, because you decide what's in and out before you see the results. Write your criteria before searching, not after. Changing criteria after seeing results introduces bias: you might unconsciously exclude studies with inconvenient findings.
[Slide 18] Step four. A protocol is a detailed plan that specifies your methods before you see the results. It includes your research question, eligibility criteria, search strategy, screening procedures, data extraction plans, quality assessment approach, and analysis methods.
[Slide 19] Why register? Registration creates a public record of what you planned to do before you knew what you'd find, which protects against accusations of cherry-picking or post-hoc changes to methods. It also prevents duplication, since other researchers can see your registered protocol and avoid working on the same question. And you can check the registry before you start, to see whether someone else is already on it.
[Slide 20] The act of writing a protocol enforces methodological discipline, by forcing you to think through your methods carefully. Decisions made in advance are more defensible than decisions made after seeing results. Many journals now require registration for systematic reviews, with some funders following suit. The registry allows you to update your registration if you need to make changes, but you must document and justify any deviations from your original protocol.
[Slide 21] Step five. Now comes the satisfying part: running your searches and watching the results pile up. But satisfying comes with a warning. This step requires meticulous documentation, or you'll regret it later.
[Slide 22] Step six. You've collected your search results, perhaps 100 citations, perhaps 10,000. Now you need to identify which ones actually meet your eligibility criteria. This happens in two stages: title and abstract screening, and then full-text screening.
[Slide 23] The gold standard is dual independent screening. Two reviewers independently screen every citation, then compare decisions and resolve disagreements through discussion or a third reviewer. This catches errors and inconsistencies, but it doubles the workload, and that's a real constraint when you're facing thousands of citations.
[Slide 24] Alternatives exist that divide the work. Once reviewers demonstrate they apply eligibility criteria consistently, some teams split the remaining citations between reviewers, with each screening independent batches. A hybrid approach has reviewers screen most citations independently but periodically screen random subsets in duplicate, to monitor ongoing agreement.
[Slide 25] If you choose an approach that divides citations between reviewers, you must first establish inter-rater reliability. Why? Because screening decisions are subjective. Reviewers bring different interpretations of eligibility criteria, different thresholds for maybe, and different levels of attention depending on fatigue. Without verification, you might have one reviewer applying stricter criteria than another, and you'd never know.
[Slide 26] So before dividing the work, pilot your process. Both reviewers screen the same 50 to 100 citations independently. Compare decisions and calculate agreement using Cohen's kappa. Discuss disagreements to clarify the eligibility criteria, and refine your screening guide as needed. Repeat until agreement is acceptable, typically a kappa above 0.8. Only then should you divide the remaining citations, and you document your approach and your inter-rater reliability in the methods section.
[Slide 27] In the first stage, you review titles and abstracts to identify citations that are potentially relevant. You're identifying which citations deserve a closer look. For each citation, ask: based on the title and abstract alone, is this study potentially about my population, intervention, comparison, and outcome? If the answer is clearly no, exclude it. If the answer is yes or maybe, include it for full-text review. This stage goes quickly once you get into a rhythm, and it would not be exceptional for an experienced reviewer to screen more than 100 abstracts per hour. But speed shouldn't compromise consistency.
[Slide 28] Steps seven and eight. Studies that survive full-text screening become your included studies, and you systematically extract information from each one into a standardized format. Then you appraise them. Not all studies are created equal. A well-designed randomized trial provides stronger evidence than a poorly designed observational study, and critical appraisal, also called risk of bias assessment, evaluates the methodological quality of each included study.
[Slide 29] Imagine two studies of the same intervention reach opposite conclusions. How do you interpret this? Critical appraisal helps you understand whether the discrepancy reflects true heterogeneity in effects, or methodological differences that make one study more trustworthy than the other. Appraisal also informs your synthesis. You might conduct sensitivity analyses excluding high-risk-of-bias studies, present results stratified by risk of bias, or downgrade your confidence in the conclusions if most of the evidence is low quality.
[Slide 30] Step nine, and it's optional. Meta-analysis is a statistical technique for combining results across studies to generate a pooled estimate of effect. It's the meta in systematic review. But meta-analysis isn't always appropriate or possible. It's appropriate when you have multiple studies measuring the same outcome, when the studies are similar enough in population, intervention, and design to meaningfully combine, when effect estimates and variance can be extracted or calculated, and when the resulting pooled estimate would answer a meaningful question.
[Slide 31] Don't force a meta-analysis when studies are too heterogeneous. Pooling across substantially different populations, interventions, or outcomes produces an estimate that may be meaningless or misleading. Don't force one when there are too few studies either; with only a handful, meta-analysis provides little benefit and can give false precision. Watch out for poor data quality: meta-analyzing low-quality studies produces a precise but potentially biased estimate. And skip it when effect measures aren't compatible, since combining odds ratios with risk ratios or hazard ratios requires assumptions that may not hold.
[Slide 32] Step ten. Your systematic review culminates in a written report that synthesizes what you found, and every systematic review needs a flow diagram showing how you went from initial search results to final included studies.
[Slide 33] The diagram tracks records identified from each source, records after deduplication, records screened on title and abstract, records excluded at screening, full-text articles assessed, full-text articles excluded with reasons, studies included in the review, and studies included in a meta-analysis if applicable. This transparency lets readers assess the comprehensiveness of your search and understand how you arrived at your included studies.