Frequently Asked Questions:
What is the Delphi Method?
The Delphi Method is a structured forecasting method used to help a group come to more accurate forecasts by soliciting anonymous forecasts in multiple rounds. It was developed by Project RAND in 1959 and is widely used in government and private enterprise. Maby automates the Delphi Method to make it easier to implement and more assessible to organizations.
How does the Delphi Method work?
(1) All participants start by independently and anonymously answering a round of forecast questions written by the moderator.
(2) Once everyone has forecast, everyone's anonymized forecasts are revealed to the group. And everyone can optionally vote for the forecasts that they find most insightful.
(3) A new round of forecasting is launched with the same questions. Participants are encouraged to update their forecasts in light of others' forecasts.
(..) This process is iterated for multiple rounds with the expectation that forcasts become more accurate and start to converge.
Why use the Delphi Method?
There are many potential benefits to using the Delphi Method.
- More honest opinions: Because forecasts are anonymous, team members are more free to express their honest opinions about sensitive topics, when their opinion is unpopular, or when they suspect their boss disagrees.
- More diverse viewpoints: Since forecasts are made before seeing what others think, a wider array of information and points of view may be expressed. This may help surface risks that managers hadn't thought of.
- Deeper Engagement: Everyone gets their voice heard on an equal footing, leading to higher engagement especially for my junior members.
- Meritocracy of ideas: Optionaly voting for insightful comments is done without knowing who wrote the comment creating a true meritocracy of ideas.
- More Accurate Forecasts: The factors above all contribute to a more meaningful and efficient discussion, helping to reach a more accurate forecast and more thoughtful business decisions.
Who is the Delphi Method for?
We believe that almost any group may be able to benefit from motivational benefits above, even without considering the improvements in forecasting. We believe that the Delphi Method may improve the accuracy of forecasts most when: (a) the question is complicated with many variables, so many people may have different insights about different variables, (b) your team is big enough that it is not practical to have everyone speak up in a meeting, (c) the topics involved are sensitive and likely to raise emotions, so there may be more fear of expressing a minority or unpopular point of view..
Does the Delphi Method really work?
The Delphi Method has been around for over half a century and continues to be used. There have been many case studies showing improved forecasts by using it. Ultimately the question of whether it leads to more accurate forecasts is empirical and can be measured. By keeping score of forecast accuracy, so you can measure if your team's forecast improves from round 1 to round 2 and beyond.
What is a forecast?
The simplest type of forecast is a probability that something will occur along with a rationale for that probability. A familiar example is the weather forecasts that meteorologists assign to the chance of rain in a few days.
What is keeping score?
By asking forecasting questions that will have an objective Yes or No answer, a probabilistic forecast can be assigned a score based on the question's outcome.
How do we keep score?
We use Brier scoring as our primary scoring technique. Brier scoring was first used by meteorologists to keep score of their weather forecasts. A forecast's Brier score is similar to the mean squared error between the forecast probability and the outcome (either 0 or 1). To ensure that people forecast their honest opinions, it is important that the scoring rule is mathematically proper.
Why keep score?
Keeping score allows us to empirically and objectively determine how to improve forecasting accuracy, ie both individuals and the group get objective feedback so that they can learn and improve. Which forecasters are more accurate? Does the Delphi Method improve forecast accuracy? Does training improve an anlyst's accuracy? Does teaming improve accuracy? Does a business process improve accuracy? All of these questions can be answered empirically by comparing average scores over many questions. For example, the scores of round 2 forecasts should be better than round 1 forecasts in the Delphi Method.
What is a binary forecast question?
A binary forecast question is a question with a Yes/No answer. For example: Will a recession start in the US in 2020?
What is a conditional forecast question?
A conditional forecast question is only scored if the condition occurs. Conditional questions allow us to forecast the relationship between two events. For example: If a recession starts in 2020, then will the S&P 500 decline more than 10% in 2020?
What is a multinomial forecast?
A multinomial forecast question has many bins for possible answers.
How do I make a forecast?
In order to make a forecast, you must first create a user account so that we can track your forecasts. Click here to Sign Up or in the upper right hand corner of this page. Once you have an account, click on Forecast at the top of this page to see the Public Challenges. Or click here.
How do I join a group?
To join a private forecasting group, you must first get the 9 digit PIN from the group's owner. Then, you can enter the PIN here or click on Forecast and then the "Join a Group" button.
How do I see my scores?
Once you've forecast at least 1 question that has resolved, you can use the "Scoreboard" button to see a summary table of your Brier scores compared against the Brier score of the Group's median forecast for each question. If you didn't forecast a round, then you are imputed the group's score for that round.
What is the leaderboard?
The "Leaderboard" button shows the ranking of group members based on average Brier score for all the resolved question rounds in that group.
How do I start my own group?
Once you are logged in, click on "Manage" in the navigation bar and the the "Create a New Group" button. Or, click here. You'll need to name your groups and choose a 9 digit PIN. Anyone who knows your PIN can join your group.
How do I ask a question?
After clicking "Manage" and then selecting your group, you should see a "New Question" button that allows you to write a new Yes/No questions.
How do I launch a round?
Once you've written a question, you can launch round 1 of that question using the "Launch Round" button. After launching round 1, members of your group will see the question listed under "Open Questions" and will be able to submit forecasts
How do I close a round?
Once you have collected forecasts from group members, use the "Close Round" button to stop accepting new forecasts. Once a round is closed, group members can see the anonymized forecasts of everyone else. And, they can optionally vote for insightful comments.
How do I create another round?
Per the Delphi Method, you can now launch a new round of forecasting on the same question. Group members are encouraged to update their forecasts based on the comments of others. You can repeat the cycle of closing and re-launching rounds.
How do I resolve a question?
Once the outcome of a quesiton is known, the question can be resolved as either "Yes" or "No". All rounds must be closed and the "Resolve Question" button can be used. An explanation of the resolution is considered good practice - so all members understand why the question resolved as it did.
The Delphi Method
Keeping Score of Forecasts
Forecasting on Maby
Managing a Group on Maby
Ready to start keeping score? Sign up for free.
Please reach out if you have any interest in forecasting! firstname.lastname@example.org