Improve your team's forecasts in 15 minutes

Shorter meetings, higher engagement, more accurate forecasts.

Contact us

Forecasting Expert vs. Disease Experts

Researchers from UMass have been collecting COVID forecasts from over 30 disease experts from insitutions like Harvard and Johns Hopkins. Tom has been forecasting the same questions in real time. The results below show all the questions, forecasts, and accuracy scores.

So far, we have been more accurate than the experts!

Forecast
(10%-90% interval)
Accuracy Score
(Brier score)
wk Question TOM EXPERTS Outcome TOM EXPERTS
AVERAGE BRIER SCORE
(green is better, red is worse)
0.194 0.247
5/10 How many cases per day will Washington state average for the week ending June 7? 170
(30-500)
331
(158-644)
155 0.289 0.713
5/10 How many confirmed US cases will there be on May 17? (in millions) 1.48
(1.45-1.57)
1.5
(1.47-1.6)
1.48 0.062 0.135
5/10 How many deaths will Pennsylvania report as of June 13? (in thousands) 7.6
(5.2-12)
7.2
(5.2-10)
6.2 0.214 0.180
5/3 What will the 7 day average new cases be in Texas for the week ending June 13? 1150
(400-3000)
1350
(750-2300)
1779 0.323 0.166
5/3 How many US COVID deaths will occur in 2020? (in thousands) 160
(90-800)
256
(118-1212)
5/3 How many confirmed US cases will there be on May 10? (in millions) 1.33
(1.27-1.39)
1.35
(1.25-1.44)
1.33 0.066 0.129
5/3 How many US states/territories will report more new cases for September compared to June? 7
(1-42)
21
(2-45)
4/26 How many confirmed US cases will there be on May 3? (in millions) 1.14
(1.09-1.21)
1.14
(1.07-1.24)
1.152 0.174 0.195
4/26 When will weekly new US deaths first fall below 5,000 (Sunday to Saturday)? 5/23
(5/9-6/26)
6/13
(5/16-7/4)
6/20 0.376 0.158
4/26 What will the average new daily cases be for Georgia between 5/10 and 5/16 inclusive? 900
(150-5000)
1044
(579-2292)
659 0.159 0.173
4/19 How many deaths due to COVID-19 will occur in the US in 2020? (in thousands) 80
(48-750)
150
(72-517)
4/19 As of May 9, how many COVID-19 related deaths will have occurred in the US? (in thousands) 56
(47-77)
70
(51-106)
73 0.420 0.170
4/19 As of May 9, how many COVID-19 related deaths will have occurred in Illinois? (in thousands) 2.4
(1.7-4.8)
2.8
(1.8-5.2)
3.3 0.294 0.170
4/19 As of May 9, how many COVID-19 related deaths will have occurred in Louisiana? (in thousands) 1.85
(1.55-3.4)
2.8
(1.85-5.0)
2.2 0.129 0.170
4/19 How many confirmed US cases will there be on April 26? (in thousands) 930
(870-990)
950
(860-1060)
959 0.222 0.127
4/12 Which of the next 6 months will see the highest total number of deaths nationwide in the US for COVID-19 illness? April
(April-June)
April
(March-July)
4/12 How many US COVID-19 deaths will occur by May 1? (in thousands) 50
(36-66)
46
(34-68)
59 0.111 0.180
4/12 How many states will report more than 1000 deaths by May 1? 12
(9-16)
9
(4-15)
14 0.193 0.712
4/12 How many US COVID-19 cases will there be on April 19? (in thousands) 755
(710-795)
780
(690-905)
751 0.093 0.151
4/12 How many US COVID-19 deaths will occur by June 1? (in thousands) 71
(48-127)
71
(45-131)
99 0.178 0.180
4/5 How many US COVID-19 cases will there be on April 12? (in thousands) 520
(470-680)
670
(510-900)
551 0.103 0.336
4/5 Which month (March-August) will see the highest total number of hospitalizations nationwide in the US for COVID-19 illness? April
(April-June)
May
(April-August)
3/29 Will NY state report fewer than 1,000 new cases on the day of April 28? 28%
(-)
50%
(-)
no 0.157 0.500
3/29 How many deaths due to COVID-19 will occur in the US in 2020? (in thousands) 240
(75-1200)
260
(80-1000)
3/29 How many US COVID-19 cases will there be on April 5? (in thousands) 410
(250-590)
420
(260-580)
332 0.123 0.145
3/29 Which month (March-August) will see the highest total number of hospitalizations nationwide in the US for COVID-19 illness? April
(April-June)
May
(April-August)
3/22 Will US COVID-19 deaths exceed 245,500 by the end of 2020? (percent probability YES) 45%
(-)
50%
(-)
3/22 Will there will be a second wave of US cases in 2020? (US active cases declines by 50% and subsequently increases by 50% AND 2nd wave must be at least 50% as high) (probability YES) 52%
(-)
73%
(-)

Click here to see how the Brier scores were calculated.


How to improve your team’s forecasts in 15 minutes

We teach your team how to implement the Delphi method using our app to make measurably more accurate forecasts. The Delphi method is a proven technique developed by the Rand Corporation in the 1950s. The basic steps are:

  1. Everyone in a group submits their anonymous forecasts to a question - so there is no group think or anchoring.
  2. Next everyone gets to see what their teammates wrote (anonymized), along with the group's initial median forecast.
  3. Participants can vote on comments they find insightful. Because comments are anonymized, voting is based on the merit of the idea, not personal politics.
  4. Everyone can update their forecasts if they read compelling comments by others. An updated group median is calculated.
  5. When the outcome of questions are known, then forecasts can be scored. Everyone can learn from the feedback and measure if the group's median forecast improved after updating.

Using our app to gather, share, and track forecasts, the above process can be completed in as little as 15 minutes for a question. While it would take 20 minutes for each member of a 20 person team to talk for a minute, our app allows everyone's comment to be read in less time - so meetings are shorter. By keeping score, you can measure the change in your team's forecast accuracy.

Key highlights of our 90 minute online sessions include:

  • Before the meeting we convert your key issues into 3-4 forecasting questions and load them onto our app.
  • During the meeting with your team, we guide your team through the Delphi method in which everyone first forecasts anonymously, then sees what everyone else thinks, votes for the best comments, and updates their forecasts.
  • We expect the Delphi method to lead to measurably better forecasts (as measured by Brier score) in as little as 15 minutes per question.
  • At the end of the meeting we’ll produce a special report containing all the forecasts, underlying data, estimates and top comments along with feedback and calibration information for everyone who participates.

Our goal is to make your team self-sufficient, so you can use the Delphi method and keep score even when we're not around.

Whether you need one-off support for urgent critical projections or are looking to build up key forecasting skills within your organisation, Maby can help!