The Delphi Method: Using Anonymous Rounds to Pool Expert Judgment

The Delphi method is a structured way to pool expert judgment. Panelists answer the same questions in private, receive an anonymous summary of the group’s answers, and answer again, usually over two or three rounds.¹ Because nobody knows who said what, an answer has to persuade on its reasons alone. To run one, you frame the questions, recruit a varied panel, fix in advance what counts as agreement, then send private rounds with feedback.

Is all that ceremony better than an ordinary meeting? In a 1999 review of mostly laboratory studies, Gene Rowe and George Wright found that Delphi panels usually beat both averaged answers and freely talking groups, though they found no consistent edge over other structured methods.² The guide to how individuals and groups reach decisions covers the other tools.

What is the Delphi method in simple terms?

At its simplest, the Delphi method is a survey you repeat. A facilitator asks a panel of experts the same questions several times, and between rounds shows everyone what the group as a whole answered and why, without names. People can then keep or change their answers, Niederberger and Spranger’s 2020 overview of the method explains.¹

The design targets a familiar problem. In a meeting, the first confident answer, the most senior person or the loudest voice tends to set the direction, and the people who doubt it go quiet; the groupthink guide covers that pattern. A Delphi run removes the room. Each expert thinks alone, so ten people produce ten independent first answers, and the group’s view is assembled on paper.

A product team estimating when a new market will be big enough to enter shows the difference. Around a table, the head of sales names a year and the rest adjust to it. In a Delphi round, a junior analyst who has seen slower adoption data writes her own year and reason, and in round two everyone reads it without knowing whose it is. Even a team that never runs a full Delphi can borrow this part: collect written answers privately before anyone speaks.

From a RAND defense project to medical guidelines

The Delphi method was developed at the RAND Corporation in the US in the 1950s, mainly by Norman Dalkey and Olaf Helmer, who described it in a 1963 paper in Management Science.³ Its first job was military: early in the Cold War, experts were asked to judge the probability, frequency and intensity of possible enemy attacks, Wikipedia’s history of the method records.⁴

Dalkey and Helmer described their method as a way to obtain “the most reliable opinion consensus of a group of experts” through a series of questionnaires with controlled feedback in between (we could reach only the abstract).⁵ In 1964 Gordon and Helmer used it for a long-range forecast covering scientific breakthroughs, automation and progress in space, and it later spread into public policy, business forecasting and health.⁴

Health researchers in many countries now use it widely.¹ Consensus methods, Delphi among them, help write clinical guidelines, define diagnostic criteria, classify diseases and set research priorities, according to the international ACCORD reporting guideline of 2024, itself built by a Delphi panel drawn from six continents.⁶ The common thread is a question that matters but lacks settled evidence, so the best answer available is the judgment of people who know the field.

Outside health the test is the same. Use Delphi for a forecast, a standard or a priority list with no data behind it yet. When a good study or a spreadsheet can give the answer, Delphi adds little.

The four features that define a Delphi study

A classic Delphi study has four defining features, condensed from five characteristics that Niederberger and Spranger list: experts answer anonymously; a standardized questionnaire is repeated over rounds; each round feeds back the group’s results so people can revise; and the group answer is calculated statistically.¹

Anonymity stops the authority, reputation or personality of some panelists from dominating others, and it makes it easier to change your mind without losing face.⁴ Iteration gives everyone a second look once they have seen the group’s view. Controlled feedback means the facilitator decides what travels between rounds: usually the group’s middle answer and spread, plus a summary of the reasons given, especially by those far from the middle.³

The statistical group response is usually the medianmedian: The middle value when numbers are put in order, so half fall below it and half above. Unlike the average (mean), it is not pulled up or down by a few very large or very small values, which is why it is often used for incomes and wealth.Full entry in the glossary and a measure of spread such as the interquartile range, the span holding the middle half of the answers. Hsu and Sandford warn that when answers cluster around two different points, a single middle value can mislead.³ A panel forecasting a product launch might split between people expecting six months and people expecting eighteen; a median of twelve describes neither camp, so report both.

1. Private answers: each panelist answers alone, and names stay hidden from the rest of the panel
2. Controlled feedback: the facilitator sends back a summary of the group's answers and the reasons behind the outlying ones
3. Revise or hold: panelists may move toward the group or keep a view they can defend; a minority answer can survive
4. Stop by the rule: the run ends when the consensus or round limit agreed before round one is reached

How many rounds are there in a Delphi study?

Most Delphi studies use two or three rounds, according to a 2020 map of 12 systematic reviews in health sciences, and Hsu and Sandford’s practical guide notes that three are often enough to reach consensus.¹³

Boulkedid and colleagues’ review of quality-indicator studies found a median of three rounds in classic Delphi studies. Stopping too soon can leave results that mean little, they noted, while too many rounds tire panelists and raise dropout, and evidence on the best number is thin.⁷

Each round costs every panelist time, so goodwill drains with every extra questionnaire. Book the panel’s time for three rounds at the outset, and agree beforehand on what would end the study early, such as answers that stop changing between rounds.

Decide what counts as consensus before round one

A vague definition of consensus is a frequent weakness in published Delphi studies. In their 2014 systematic reviewsystematic review: A review that fixes its question and its rules for including studies in advance, then searches out every study that fits and weighs them together. Some systematic reviews pool the results into a meta-analysis; others describe what the studies found without combining the numbers.Full entry in the glossary of a random sample of 100 Delphi studies, Ian Diamond and colleagues in Canada found that most claimed to measure consensus but only 72 defined it, and the most common definition, percent agreement, had a median threshold of 75 percent (this rests on the abstract alone).⁸

Why does this matter? A threshold chosen after the data are in can be set wherever the results happen to fall. Diamond’s team found that consensus decided when a study stopped in only a minority of cases; far more simply ended after a planned number of rounds.⁸ In many studies, then, the reported result may only show where the group stood when the planned rounds ran out.

A good rule names three things before anyone answers: the measure (for example, the share of panelists rating an item 7 to 9 on a 9-point scale), the cut-off, and what happens to items that miss it; Boulkedid’s review describes rules of this kind.⁷ Write the rule into the invitation, so panelists know what their ratings will decide.

How to run a Delphi study, step by step

Running a Delphi study comes down to six steps: frame the question, choose the panel, set the consensus rule, run round one, feed back and repeat, then report. They draw on Hsu and Sandford’s guide and Boulkedid’s review.

1. Frame questions the panel can answer

Ask for judgments, such as a date, a probability, a rating of importance or a ranked list, and give everyone the same short brief of the facts available. A first round can be open-ended to collect ideas, which the facilitator then turns into statements to rate.³

2. Choose panelists for knowledge and range

Pick people for what they know, and include different vantage points: Boulkedid’s review recommends a panel that covers the stakeholders who have an interest in the result, and asking people to commit to every round before you start.⁷ There is no agreed panel size. Hsu and Sandford cite advice that ten to fifteen people can be enough if their backgrounds are similar, and the ACCORD authors note that panels of 20 to 30 are common.³⁶

3. Fix the consensus rule and the stopping rule

Write down the measure, the threshold and the maximum number of rounds. Agree how you will report items that never reach consensus, because a lasting disagreement is a finding too.

4. Run round one in private

Send the questionnaire, collect answers and reasons, and keep identities hidden from other panelists. Allow about two weeks per round; Hsu and Sandford cite a recommended minimum of 45 days for a whole study.³

5. Feed back the median, the spread and the reasons

Send each panelist the group’s median and range, their own previous answer and a summary of the comments, as Boulkedid’s team recommends, then ask for a new answer.⁷ Ask people far from the middle to explain their view, since their reasons often matter most to the next round.

6. Stop by your rule and report everything

End the study when your rule says so, and report the panel’s make-up, the response rate in each round, the consensus definition and the items that failed. ACCORD lists missing panel backgrounds, consensus definitions and per-round response rates among common gaps in published reports.⁶

A logistics company wants a forecast of how many weeks its warehouse software migration will take. A facilitator recruits nine people from IT, three warehouse sites and the software vendor, and agrees in advance to stop when the middle half of answers spans four weeks or less, or after round three. The numbers below are illustrative.

The median rose because two reasons persuaded people; the outliers did not just drift to the middle. If round one had been a meeting, the IT lead’s first estimate might have anchored everyone, and the two delays would have surfaced only once the project slipped. The same private-estimate-then-reveal logic, run in minutes rather than weeks, is what software teams use in planning poker.

Before you send round one

• Write each question so it asks for a judgment
• Give every panelist the same short brief
• Recruit for knowledge and for different vantage points
• Ask panelists to commit to every round
• Fix the consensus measure, threshold and round limit
• Decide how to report items without consensus

Does the Delphi method beat a meeting?

In laboratory tests, usually yes. Rowe and Wright’s 1999 review of mostly laboratory studies found that Delphi panels mostly outscored simple averages of separate answers and groups that talked freely, with no consistent advantage over other structured group techniques.²

The study

Limited evidence

How Delphi panels fared against averages and open discussion

Against statistical groups, where separate answers are simply averaged, Delphi did better in 12 studies, worse in two and tied in two. Against standard interacting groups, which discussed the question in the usual way, it did better in five studies, worse in one and tied in two. Against other structured group methods, the evidence showed no consistent winner.²

The authors added a large caveat: the lab versions often strayed from the original design, with little control over how expert the panelists were or what feedback they saw, so it is hard to generalize about Delphi as a whole. A Delphi run to its ideal specification, they argued, might do better.²

Agreement also differs from accuracy. A Delphi consensus shows that experts agreed, which does not by itself show they were right, and if the panel is misinformed, the rounds may only make its error look more confident.⁴ Ten specialists who all read the same flawed forecast can reach a tidy consensus around it. In the evidence ranking of the former US Agency for Health Care Policy and Research, Delphi results sit at the bottom for judging cause and effect, below trials and correlation studies.¹

What is the difference between Delphi and a focus group?

A focus group gathers people in one discussion to explore views in depth, while a classic Delphi panel never meets and works through anonymous, repeated questionnaires toward a measurable result.⁴

In a face-to-face group, strong personalities can steer the conversation. A Delphi panel loses the clarifying back-and-forth that a meeting allows, which is why many health studies add a structured meeting between or after rounds, a hybrid usually called a modified Delphi.⁷ A city council asking residents what they want from a new park would run focus groups; the same council asking traffic engineers how many years an old bridge has left would run a Delphi. Choose a focus group to learn why people think what they think, and a Delphi when you need a defensible group answer from experts who may be spread across countries.

Advantages and disadvantages of the Delphi technique

The main advantages of the Delphi technique are independent first answers, equal weight for every panelist and a written record of reasons; the main disadvantages are time, dependence on the panel and loose definitions of consensus.

Time needs the most planning. Among the quality-indicator studies Boulkedid’s team reviewed, the ones that reported duration took a median of about 20 weeks, probably because the facilitator must chase late responses and analyze each round before writing the next.⁷ For a decision needed by Friday, a well-run meeting with silent writing first is the better tool.

The bottom line

The Delphi method earns its slowness when experts disagree, the question has no settled evidence, and rank or confidence would sway a meeting. Before round one, choose the panel for knowledge and range, write down what counts as consensus and when you will stop, and plan to report the disagreements as carefully as the agreements.