How do you turn distributed expertise into consensus?
Indicators to monitor pandemic preparedness in low-resource settings
The evidence can be reviewed, but ultimately it's a judgment call
The experts are distributed across continents
Slide 3
Delphi studies
A Delphi study runs in rounds
Questions can be open-ended or rating-based. The process iterates until ratings stabilize or a predefined consensus threshold is met.
Slide 4
Delphi studies
Anonymity and controlled feedback
Anonymity blunts the deference, hierarchy, and personality dynamics of meetings
Controlled feedback lets experts update on the group's view each round
No pressure to capitulate face to face
Slide 5
Delphi studies
When a Delphi study is a strong choice
An empirical answer doesn't exist and won't soon
Stakeholders span geographies
A decision needs the visible legitimacy of a transparent process
Slide 6
Delphi studies
Strengths and limitations of Delphi studies
Relatively cheap, scales geographically, and leaves an audit trail
The consensus embeds whatever biases the expert panel brings
Convergence isn't the same as correctness
Report panel composition and recruitment like eligibility criteria
Delphi studies
Slide 1The Delphi method is a structured process for eliciting expert judgment and progressively narrowing toward consensus.
Delphi studies
How do you turn distributed expertise into consensus?
Indicators to monitor pandemic preparedness in low-resource settings
The evidence can be reviewed, but ultimately it's a judgment call
The experts are distributed across continents
Slide 2Imagine you're tasked with building a set of indicators to monitor pandemic preparedness in low-resource settings. You can search the literature and review the evidence, but ultimately it's a judgment call. You need experts who've spent decades on outbreak response, and they're distributed across continents. How do you turn that distributed expertise into consensus? The Delphi method is used heavily in global health for setting research priorities, developing indicators, agreeing on case definitions, and building clinical guidelines when empirical evidence is sparse.
Delphi studies
A Delphi study runs in rounds
Questions can be open-ended or rating-based. The process iterates until ratings stabilize or a predefined consensus threshold is met.
Slide 3A Delphi runs in rounds. Round one asks experts a question independently and anonymously, and the research team aggregates the responses. Round two shows each expert the group's aggregated answer and asks them to rate or rank again, with the option to revise. The process iterates over two to four rounds, until ratings stabilize or a predefined consensus threshold is met.
Delphi studies
Anonymity and controlled feedback
Anonymity blunts the deference, hierarchy, and personality dynamics of meetings
Controlled feedback lets experts update on the group's view each round
No pressure to capitulate face to face
Slide 4Two features make this more than a fancy survey. Anonymity blunts the deference, hierarchy, and personality dynamics that derail in-person consensus meetings. Controlled feedback lets participants update on the group's view each round without being pressured to capitulate face to face.
Delphi studies
When a Delphi study is a strong choice
An empirical answer doesn't exist and won't soon
Stakeholders span geographies
A decision needs the visible legitimacy of a transparent process
Slide 5Delphi is a strong choice when an empirical answer doesn't exist and won't soon, when the judgment we need is genuinely the aggregate of expert opinion, when stakeholders span geographies, or when a decision needs the visible legitimacy of asking thirty experts in a transparent process, and not three people we know. Priority-setting for child-health research and indicator development for the Sustainable Development Goals both use Delphi-family approaches at their core.
Delphi studies
Strengths and limitations of Delphi studies
Relatively cheap, scales geographically, and leaves an audit trail
The consensus embeds whatever biases the expert panel brings
Convergence isn't the same as correctness
Report panel composition and recruitment like eligibility criteria
Slide 6Delphi turns scattered expert judgment into a documented process. It's relatively cheap, scales geographically, and produces an audit trail of how consensus was reached. It also embeds whatever biases the expert panel brings. A panel that's, say, 80% high-income-country researchers will produce a consensus that reflects high-income-country priorities, no matter how rigorous the process. A Delphi can converge on a wrong answer if every expert shares the same blind spot, and the choice of who counts as an expert is itself a judgment call. Report panel composition and recruitment logic with the same transparency you'd give a study's eligibility criteria.